Nov 02 22:49:42.038361 ip-10-0-130-211 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 02 22:49:42.038375 ip-10-0-130-211 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 02 22:49:42.038384 ip-10-0-130-211 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 02 22:49:42.038682 ip-10-0-130-211 systemd[1]: Failed to start Kubernetes Kubelet. Nov 02 22:49:52.046498 ip-10-0-130-211 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 02 22:49:52.046514 ip-10-0-130-211 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 5a4a1bc2bce44bfd961e8ec63843ac65 -- Nov 02 22:51:46.942144 ip-10-0-130-211 systemd[1]: Starting Kubernetes Kubelet... Nov 02 22:51:47.342192 ip-10-0-130-211 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:51:47.342192 ip-10-0-130-211 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 22:51:47.342192 ip-10-0-130-211 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:51:47.342192 ip-10-0-130-211 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:51:47.347360 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347279 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:51:47.347360 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347360 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:51:47.347360 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347364 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:51:47.347458 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347367 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:51:47.347458 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347371 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:51:47.347458 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347374 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:51:47.347458 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347377 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:51:47.347458 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347379 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:51:47.347458 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347382 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:51:47.347458 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347385 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:51:47.347458 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347388 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:51:47.347458 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347390 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:51:47.347458 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347393 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:51:47.347458 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347396 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:51:47.347458 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347398 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:51:47.347458 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347401 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:51:47.347458 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347403 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:51:47.347458 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347414 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:51:47.347458 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347417 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:51:47.347458 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347420 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:51:47.347458 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347423 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:51:47.347458 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347426 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:51:47.347458 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347428 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:51:47.347934 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347431 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:51:47.347934 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347434 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:51:47.347934 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347437 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:51:47.347934 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347440 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:51:47.347934 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347442 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:51:47.347934 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347445 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:51:47.347934 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347447 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:51:47.347934 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347450 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:51:47.347934 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347452 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:51:47.347934 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347455 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:51:47.347934 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347457 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:51:47.347934 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347460 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:51:47.347934 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347463 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:51:47.347934 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347466 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:51:47.347934 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347470 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:51:47.347934 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347472 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:51:47.347934 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347475 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:51:47.347934 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347478 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:51:47.347934 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347481 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:51:47.347934 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347483 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:51:47.348413 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347486 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:51:47.348413 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347490 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:51:47.348413 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347494 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:51:47.348413 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347497 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:51:47.348413 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347500 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:51:47.348413 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347502 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:51:47.348413 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347505 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:51:47.348413 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347507 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:51:47.348413 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347510 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:51:47.348413 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347513 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:51:47.348413 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347515 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:51:47.348413 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347518 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:51:47.348413 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347521 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:51:47.348413 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347524 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:51:47.348413 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347527 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:51:47.348413 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347530 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:51:47.348413 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347532 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:51:47.348413 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347536 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:51:47.348413 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347539 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:51:47.348413 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347541 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:51:47.348901 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347544 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:51:47.348901 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347546 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:51:47.348901 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347549 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:51:47.348901 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347551 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:51:47.348901 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347554 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:51:47.348901 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347557 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:51:47.348901 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347559 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:51:47.348901 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347562 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:51:47.348901 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347565 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:51:47.348901 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347568 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:51:47.348901 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347571 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:51:47.348901 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347574 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:51:47.348901 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347576 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:51:47.348901 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347579 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:51:47.348901 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347581 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:51:47.348901 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347584 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:51:47.348901 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347587 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:51:47.348901 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347589 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:51:47.348901 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347592 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:51:47.348901 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347595 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:51:47.349382 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347599 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:51:47.349382 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347603 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:51:47.349382 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347607 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:51:47.349382 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347610 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:51:47.349382 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347613 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:51:47.349382 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347616 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:51:47.349382 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347619 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:51:47.349382 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347622 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:51:47.349382 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347625 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:51:47.349382 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347627 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:51:47.349382 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347629 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:51:47.349382 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347632 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:51:47.349382 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347635 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:51:47.349382 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347637 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:51:47.349382 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347640 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:51:47.349382 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347642 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:51:47.349382 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347645 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:51:47.349382 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347647 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:51:47.349841 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347650 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:51:47.349841 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347652 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:51:47.349841 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347655 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:51:47.349841 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347657 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:51:47.349841 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.347660 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:51:47.349841 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348484 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:51:47.349841 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348492 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:51:47.349841 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348494 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:51:47.349841 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348497 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:51:47.349841 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348500 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:51:47.349841 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348503 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:51:47.349841 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348506 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:51:47.349841 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348509 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:51:47.349841 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348512 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:51:47.349841 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348514 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:51:47.349841 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348517 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:51:47.349841 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348520 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:51:47.349841 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348522 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:51:47.349841 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348525 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:51:47.349841 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348528 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:51:47.350355 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348531 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:51:47.350355 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348533 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:51:47.350355 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348536 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:51:47.350355 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348539 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:51:47.350355 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348541 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:51:47.350355 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348544 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:51:47.350355 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348546 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:51:47.350355 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348551 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:51:47.350355 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348555 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:51:47.350355 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348558 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:51:47.350355 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348561 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:51:47.350355 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348564 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:51:47.350355 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348566 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:51:47.350355 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348569 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:51:47.350355 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348571 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:51:47.350355 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348574 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:51:47.350355 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348576 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:51:47.350355 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348580 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:51:47.350355 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348582 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:51:47.350817 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348585 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:51:47.350817 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348588 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:51:47.350817 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348590 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:51:47.350817 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348593 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:51:47.350817 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348596 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:51:47.350817 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348598 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:51:47.350817 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348601 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:51:47.350817 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348603 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:51:47.350817 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348606 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:51:47.350817 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348609 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:51:47.350817 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348612 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:51:47.350817 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348614 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:51:47.350817 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348617 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:51:47.350817 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348619 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:51:47.350817 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348622 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:51:47.350817 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348625 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:51:47.350817 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348627 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:51:47.350817 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348629 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:51:47.350817 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348632 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:51:47.350817 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348634 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:51:47.351299 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348637 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:51:47.351299 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348639 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:51:47.351299 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348642 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:51:47.351299 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348644 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:51:47.351299 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348646 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:51:47.351299 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348649 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:51:47.351299 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348652 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:51:47.351299 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348654 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:51:47.351299 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348657 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:51:47.351299 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348659 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:51:47.351299 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348661 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:51:47.351299 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348664 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:51:47.351299 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348667 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:51:47.351299 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348669 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:51:47.351299 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348672 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:51:47.351299 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348675 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:51:47.351299 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348678 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:51:47.351299 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348681 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:51:47.351299 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348683 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:51:47.351299 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348686 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:51:47.351762 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348688 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:51:47.351762 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348693 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:51:47.351762 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348695 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:51:47.351762 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348699 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:51:47.351762 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348703 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:51:47.351762 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348706 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:51:47.351762 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348709 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:51:47.351762 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348712 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:51:47.351762 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348714 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:51:47.351762 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348717 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:51:47.351762 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348719 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:51:47.351762 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348722 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:51:47.351762 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348724 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:51:47.351762 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348727 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:51:47.351762 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348730 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:51:47.351762 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348732 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:51:47.351762 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348735 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:51:47.351762 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348737 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:51:47.351762 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348740 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:51:47.352229 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348742 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:51:47.352229 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348745 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:51:47.352229 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348747 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:51:47.352229 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348750 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:51:47.352229 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348752 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:51:47.352229 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348756 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:51:47.352229 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348759 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:51:47.352229 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348762 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:51:47.352229 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348765 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:51:47.352229 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348768 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:51:47.352229 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348770 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:51:47.352229 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348773 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:51:47.352229 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.348775 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:51:47.352229 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349370 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:51:47.352229 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349377 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:51:47.352229 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349381 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:51:47.352229 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349384 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:51:47.352229 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349388 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:51:47.352229 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349390 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:51:47.352229 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349393 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:51:47.352704 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349396 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:51:47.352704 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349399 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:51:47.352704 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349402 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:51:47.352704 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349404 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:51:47.352704 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349407 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:51:47.352704 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349409 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:51:47.352704 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349418 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:51:47.352704 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349421 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:51:47.352704 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349424 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:51:47.352704 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349428 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:51:47.352704 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349431 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:51:47.352704 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349434 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:51:47.352704 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349436 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:51:47.352704 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349439 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:51:47.352704 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349441 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:51:47.352704 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349444 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:51:47.352704 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349447 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:51:47.352704 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349455 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:51:47.352704 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349458 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:51:47.352704 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349460 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:51:47.353208 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349463 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:51:47.353208 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349466 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:51:47.353208 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349469 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:51:47.353208 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349471 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:51:47.353208 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349474 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:51:47.353208 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349477 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:51:47.353208 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349479 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:51:47.353208 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349482 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:51:47.353208 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349485 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:51:47.353208 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349488 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:51:47.353208 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349490 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:51:47.353208 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349493 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:51:47.353208 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349496 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:51:47.353208 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349498 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:51:47.353208 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349501 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:51:47.353208 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349504 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:51:47.353208 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349506 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:51:47.353208 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349509 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:51:47.353208 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349511 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:51:47.353661 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349514 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:51:47.353661 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349516 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:51:47.353661 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349520 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:51:47.353661 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349523 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:51:47.353661 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349525 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:51:47.353661 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349528 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:51:47.353661 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349530 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:51:47.353661 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349533 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:51:47.353661 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349535 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:51:47.353661 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349538 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:51:47.353661 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349540 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:51:47.353661 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349549 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:51:47.353661 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349552 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:51:47.353661 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349554 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:51:47.353661 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349557 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:51:47.353661 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349560 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:51:47.353661 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349562 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:51:47.353661 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349565 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:51:47.353661 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349567 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:51:47.354128 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349571 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:51:47.354128 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349575 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:51:47.354128 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349578 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:51:47.354128 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349581 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:51:47.354128 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349583 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:51:47.354128 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349586 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:51:47.354128 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349589 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:51:47.354128 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349591 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:51:47.354128 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349593 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:51:47.354128 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349596 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:51:47.354128 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349598 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:51:47.354128 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349601 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:51:47.354128 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349604 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:51:47.354128 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349608 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:51:47.354128 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349611 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:51:47.354128 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349617 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:51:47.354128 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349620 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:51:47.354128 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349623 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:51:47.354128 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349625 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:51:47.354607 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349628 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:51:47.354607 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349630 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:51:47.354607 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349633 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:51:47.354607 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349635 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:51:47.354607 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349638 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:51:47.354607 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349640 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:51:47.354607 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349650 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:51:47.354607 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349653 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:51:47.354607 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349658 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:51:47.354607 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349660 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:51:47.354607 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349663 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:51:47.354607 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349665 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:51:47.354607 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349668 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:51:47.354607 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349670 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:51:47.354607 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349673 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:51:47.354607 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349675 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:51:47.354607 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349678 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:51:47.354607 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349680 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:51:47.354607 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349683 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:51:47.355068 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349685 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:51:47.355068 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349688 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:51:47.355068 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.349690 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:51:47.355068 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.350829 2575 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 22:51:47.355068 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.350840 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 22:51:47.355068 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.350851 2575 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 22:51:47.355068 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.350856 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 22:51:47.355068 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.350875 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 22:51:47.355068 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.350879 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 22:51:47.355068 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.350884 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 22:51:47.355068 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.350888 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 22:51:47.355068 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.350892 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 22:51:47.355068 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.350895 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 22:51:47.355068 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.350899 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 22:51:47.355068 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.350902 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 22:51:47.355068 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.350905 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 22:51:47.355068 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.350908 2575 flags.go:64] FLAG: --cgroup-root="" Nov 02 22:51:47.355068 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.350911 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 22:51:47.355068 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.350914 2575 flags.go:64] FLAG: --client-ca-file="" Nov 02 22:51:47.355068 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.350917 2575 flags.go:64] FLAG: --cloud-provider="external" Nov 02 22:51:47.355068 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.350926 2575 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 22:51:47.355068 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.350934 2575 flags.go:64] FLAG: --cluster-domain="" Nov 02 22:51:47.355068 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.350937 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 22:51:47.355690 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.350940 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 22:51:47.355690 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.350944 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 22:51:47.355690 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.350947 2575 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 22:51:47.355690 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.350952 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 22:51:47.355690 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.350955 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 22:51:47.355690 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.350958 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 22:51:47.355690 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.350961 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 22:51:47.355690 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.350964 2575 flags.go:64] FLAG: --contention-profiling="false" Nov 02 22:51:47.355690 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.350967 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 22:51:47.355690 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.350970 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 22:51:47.355690 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.350974 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 22:51:47.355690 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.350977 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 22:51:47.355690 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.350981 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 22:51:47.355690 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.350984 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 22:51:47.355690 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.350987 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 22:51:47.355690 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.350990 2575 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 22:51:47.355690 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.350993 2575 flags.go:64] FLAG: --enable-server="true" Nov 02 22:51:47.355690 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.350996 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 22:51:47.355690 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351003 2575 flags.go:64] FLAG: --event-burst="100" Nov 02 22:51:47.355690 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351007 2575 flags.go:64] FLAG: --event-qps="50" Nov 02 22:51:47.355690 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351010 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 22:51:47.355690 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351013 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 22:51:47.355690 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351016 2575 flags.go:64] FLAG: --eviction-hard="" Nov 02 22:51:47.355690 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351020 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 22:51:47.356294 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351023 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 22:51:47.356294 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351026 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 22:51:47.356294 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351030 2575 flags.go:64] FLAG: --eviction-soft="" Nov 02 22:51:47.356294 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351033 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 22:51:47.356294 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351035 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 22:51:47.356294 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351038 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 22:51:47.356294 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351047 2575 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 22:51:47.356294 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351051 2575 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 22:51:47.356294 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351054 2575 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 22:51:47.356294 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351056 2575 flags.go:64] FLAG: --feature-gates="" Nov 02 22:51:47.356294 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351061 2575 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 22:51:47.356294 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351064 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 22:51:47.356294 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351067 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 22:51:47.356294 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351070 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 22:51:47.356294 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351073 2575 flags.go:64] FLAG: --healthz-port="10248" Nov 02 22:51:47.356294 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351076 2575 flags.go:64] FLAG: --help="false" Nov 02 22:51:47.356294 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351079 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:51:47.356294 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351083 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 22:51:47.356294 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351086 2575 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 22:51:47.356294 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351089 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 22:51:47.356294 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351092 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 22:51:47.356294 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351096 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 22:51:47.356294 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351098 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 22:51:47.356899 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351101 2575 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 22:51:47.356899 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351104 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 22:51:47.356899 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351107 2575 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 22:51:47.356899 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351110 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 22:51:47.356899 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351113 2575 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 22:51:47.356899 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351116 2575 flags.go:64] FLAG: --kube-reserved="" Nov 02 22:51:47.356899 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351119 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 22:51:47.356899 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351132 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 22:51:47.356899 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351137 2575 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 22:51:47.356899 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351140 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 22:51:47.356899 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351143 2575 flags.go:64] FLAG: --lock-file="" Nov 02 22:51:47.356899 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351146 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 22:51:47.356899 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351150 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 22:51:47.356899 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351153 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 22:51:47.356899 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351159 2575 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 22:51:47.356899 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351162 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 22:51:47.356899 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351171 2575 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 22:51:47.356899 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351175 2575 flags.go:64] FLAG: --logging-format="text" Nov 02 22:51:47.356899 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351178 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 22:51:47.356899 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351181 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 22:51:47.356899 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351184 2575 flags.go:64] FLAG: --manifest-url="" Nov 02 22:51:47.356899 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351187 2575 flags.go:64] FLAG: --manifest-url-header="" Nov 02 22:51:47.356899 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351192 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 22:51:47.356899 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351195 2575 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 22:51:47.356899 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351199 2575 flags.go:64] FLAG: --max-pods="110" Nov 02 22:51:47.357472 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351202 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 22:51:47.357472 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351205 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 22:51:47.357472 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351208 2575 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 22:51:47.357472 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351211 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 22:51:47.357472 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351214 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 22:51:47.357472 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351217 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 22:51:47.357472 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351220 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 22:51:47.357472 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351227 2575 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 22:51:47.357472 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351230 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 22:51:47.357472 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351234 2575 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 22:51:47.357472 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351237 2575 flags.go:64] FLAG: --pod-cidr="" Nov 02 22:51:47.357472 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351240 2575 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 22:51:47.357472 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351243 2575 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 22:51:47.357472 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351246 2575 flags.go:64] FLAG: --pods-per-core="0" Nov 02 22:51:47.357472 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351248 2575 flags.go:64] FLAG: --port="10250" Nov 02 22:51:47.357472 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351252 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 22:51:47.357472 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351255 2575 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-06dbd8718a6f823b3" Nov 02 22:51:47.357472 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351258 2575 flags.go:64] FLAG: --qos-reserved="" Nov 02 22:51:47.357472 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351261 2575 flags.go:64] FLAG: --read-only-port="10255" Nov 02 22:51:47.357472 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351264 2575 flags.go:64] FLAG: --register-node="true" Nov 02 22:51:47.357472 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351267 2575 flags.go:64] FLAG: --register-with-taints="" Nov 02 22:51:47.357472 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351271 2575 flags.go:64] FLAG: --registry-burst="10" Nov 02 22:51:47.357472 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351274 2575 flags.go:64] FLAG: --registry-qps="5" Nov 02 22:51:47.357472 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351276 2575 flags.go:64] FLAG: --reserved-cpus="" Nov 02 22:51:47.357472 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351286 2575 flags.go:64] FLAG: --reserved-memory="" Nov 02 22:51:47.357472 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351290 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 22:51:47.358157 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351296 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 22:51:47.358157 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351299 2575 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 22:51:47.358157 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351302 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 22:51:47.358157 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351305 2575 flags.go:64] FLAG: --runonce="false" Nov 02 22:51:47.358157 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351308 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 22:51:47.358157 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351312 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 22:51:47.358157 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351315 2575 flags.go:64] FLAG: --seccomp-default="false" Nov 02 22:51:47.358157 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351318 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 22:51:47.358157 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351321 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 22:51:47.358157 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351324 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 22:51:47.358157 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351328 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 22:51:47.358157 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351331 2575 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 22:51:47.358157 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351334 2575 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 22:51:47.358157 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351336 2575 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 22:51:47.358157 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351339 2575 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 22:51:47.358157 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351342 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 22:51:47.358157 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351345 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 22:51:47.358157 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351348 2575 flags.go:64] FLAG: --system-cgroups="" Nov 02 22:51:47.358157 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351354 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 22:51:47.358157 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351360 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 22:51:47.358157 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351366 2575 flags.go:64] FLAG: --tls-cert-file="" Nov 02 22:51:47.358157 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351369 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 22:51:47.358157 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351383 2575 flags.go:64] FLAG: --tls-min-version="" Nov 02 22:51:47.358157 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351386 2575 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 22:51:47.358157 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351389 2575 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 22:51:47.358737 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351392 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 22:51:47.358737 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351395 2575 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 22:51:47.358737 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351398 2575 flags.go:64] FLAG: --v="2" Nov 02 22:51:47.358737 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351403 2575 flags.go:64] FLAG: --version="false" Nov 02 22:51:47.358737 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351407 2575 flags.go:64] FLAG: --vmodule="" Nov 02 22:51:47.358737 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351415 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 22:51:47.358737 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351424 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 22:51:47.358737 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351622 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:51:47.358737 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351626 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:51:47.358737 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351630 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:51:47.358737 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351633 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:51:47.358737 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351635 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:51:47.358737 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351638 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:51:47.358737 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351641 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:51:47.358737 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351643 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:51:47.358737 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351646 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:51:47.358737 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351648 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:51:47.358737 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351651 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:51:47.358737 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351653 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:51:47.358737 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351656 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:51:47.358737 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351658 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:51:47.359261 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351661 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:51:47.359261 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351663 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:51:47.359261 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351666 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:51:47.359261 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351669 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:51:47.359261 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351671 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:51:47.359261 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351674 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:51:47.359261 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351678 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:51:47.359261 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351680 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:51:47.359261 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351683 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:51:47.359261 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351686 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:51:47.359261 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351688 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:51:47.359261 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351691 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:51:47.359261 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351693 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:51:47.359261 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351696 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:51:47.359261 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351699 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:51:47.359261 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351701 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:51:47.359261 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351704 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:51:47.359261 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351706 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:51:47.359261 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351716 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:51:47.359261 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351718 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:51:47.359732 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351721 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:51:47.359732 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351724 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:51:47.359732 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351726 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:51:47.359732 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351729 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:51:47.359732 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351731 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:51:47.359732 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351734 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:51:47.359732 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351737 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:51:47.359732 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351739 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:51:47.359732 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351742 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:51:47.359732 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351744 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:51:47.359732 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351747 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:51:47.359732 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351749 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:51:47.359732 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351752 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:51:47.359732 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351754 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:51:47.359732 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351757 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:51:47.359732 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351759 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:51:47.359732 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351761 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:51:47.359732 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351764 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:51:47.359732 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351768 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:51:47.360202 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351771 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:51:47.360202 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351774 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:51:47.360202 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351777 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:51:47.360202 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351779 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:51:47.360202 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351782 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:51:47.360202 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351784 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:51:47.360202 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351787 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:51:47.360202 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351789 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:51:47.360202 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351793 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:51:47.360202 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351797 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:51:47.360202 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351806 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:51:47.360202 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351810 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:51:47.360202 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351821 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:51:47.360202 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351824 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:51:47.360202 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351827 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:51:47.360202 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351829 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:51:47.360202 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351832 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:51:47.360202 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351834 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:51:47.360202 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351837 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:51:47.360202 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351839 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:51:47.360680 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351842 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:51:47.360680 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351845 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:51:47.360680 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351847 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:51:47.360680 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351850 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:51:47.360680 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351852 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:51:47.360680 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351855 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:51:47.360680 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351857 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:51:47.360680 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351873 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:51:47.360680 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351876 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:51:47.360680 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351879 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:51:47.360680 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351883 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:51:47.360680 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351888 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:51:47.360680 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351891 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:51:47.360680 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351894 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:51:47.360680 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351897 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:51:47.360680 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351900 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:51:47.360680 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351902 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:51:47.360680 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351905 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:51:47.360680 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351907 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:51:47.361140 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351910 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:51:47.361140 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351913 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:51:47.361140 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351915 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:51:47.361140 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351918 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:51:47.361140 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351922 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:51:47.361140 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351925 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:51:47.361140 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351933 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:51:47.361140 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351936 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:51:47.361140 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351939 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:51:47.361140 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351942 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:51:47.361140 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351944 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:51:47.361140 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351947 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:51:47.361140 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351949 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:51:47.361140 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.351952 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:51:47.361140 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.351978 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:51:47.361140 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352021 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:51:47.361524 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352026 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:51:47.361524 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352029 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:51:47.361524 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352032 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:51:47.361524 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352034 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:51:47.361524 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352037 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:51:47.361524 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352040 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:51:47.361524 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352042 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:51:47.361524 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352045 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:51:47.361524 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352049 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:51:47.361524 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352054 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:51:47.361524 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352057 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:51:47.361524 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352059 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:51:47.361524 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352063 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:51:47.361524 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352066 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:51:47.361524 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352068 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:51:47.361524 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352071 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:51:47.361524 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352073 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:51:47.361524 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352076 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:51:47.361524 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352078 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:51:47.361981 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352081 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:51:47.361981 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352085 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:51:47.361981 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352088 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:51:47.361981 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352090 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:51:47.361981 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352094 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:51:47.361981 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352096 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:51:47.361981 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352099 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:51:47.361981 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352102 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:51:47.361981 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352104 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:51:47.361981 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352106 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:51:47.361981 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352109 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:51:47.361981 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352112 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:51:47.361981 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352114 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:51:47.361981 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352117 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:51:47.361981 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352119 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:51:47.361981 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352122 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:51:47.361981 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352124 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:51:47.361981 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352127 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:51:47.361981 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352129 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:51:47.362422 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352132 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:51:47.362422 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352134 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:51:47.362422 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352137 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:51:47.362422 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352141 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:51:47.362422 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352144 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:51:47.362422 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352146 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:51:47.362422 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352149 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:51:47.362422 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352151 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:51:47.362422 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352154 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:51:47.362422 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352157 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:51:47.362422 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352159 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:51:47.362422 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352162 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:51:47.362422 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352164 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:51:47.362422 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352167 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:51:47.362422 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352170 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:51:47.362422 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352173 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:51:47.362422 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352176 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:51:47.362422 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352178 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:51:47.362422 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352182 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:51:47.362422 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352185 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:51:47.362897 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352188 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:51:47.362897 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352190 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:51:47.362897 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352192 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:51:47.362897 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352195 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:51:47.362897 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352198 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:51:47.362897 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352200 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:51:47.362897 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352203 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:51:47.362897 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352206 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:51:47.362897 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352208 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:51:47.362897 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352211 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:51:47.362897 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352213 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:51:47.362897 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352216 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:51:47.362897 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352219 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:51:47.362897 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352221 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:51:47.362897 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352224 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:51:47.362897 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352228 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:51:47.362897 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352230 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:51:47.362897 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352233 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:51:47.362897 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352236 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:51:47.362897 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352239 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:51:47.363496 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352241 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:51:47.363496 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352244 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:51:47.363496 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352247 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:51:47.363496 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352249 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:51:47.363496 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352252 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:51:47.363496 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352255 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:51:47.363496 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352258 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:51:47.363496 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352262 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:51:47.363496 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352264 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:51:47.363496 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352266 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:51:47.363496 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352269 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:51:47.363496 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352272 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:51:47.363496 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352275 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:51:47.363496 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352277 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:51:47.363496 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352281 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:51:47.363496 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352284 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:51:47.363496 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352287 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:51:47.363496 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352290 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:51:47.363496 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352293 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:51:47.363496 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352296 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:51:47.364559 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352298 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:51:47.364559 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352301 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:51:47.364559 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352303 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:51:47.364559 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352306 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:51:47.364559 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352309 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:51:47.364559 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352311 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:51:47.364559 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.352314 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:51:47.364559 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.359613 2575 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 22:51:47.364559 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.359630 2575 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 22:51:47.364559 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.359647 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 22:51:47.364559 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.359651 2575 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 22:51:47.364559 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359703 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:51:47.364559 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359708 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:51:47.364559 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359711 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:51:47.364559 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359714 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:51:47.364559 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359716 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:51:47.364559 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359719 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:51:47.364559 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359722 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:51:47.364559 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359725 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:51:47.365224 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359728 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:51:47.365224 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359731 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:51:47.365224 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359733 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:51:47.365224 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359736 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:51:47.365224 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359738 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:51:47.365224 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359741 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:51:47.365224 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359744 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:51:47.365224 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359746 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:51:47.365224 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359749 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:51:47.365224 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359752 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:51:47.365224 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359754 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:51:47.365224 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359757 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:51:47.365224 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359759 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:51:47.365224 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359762 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:51:47.365224 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359764 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:51:47.365224 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359767 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:51:47.365224 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359769 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:51:47.365224 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359772 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:51:47.365224 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359774 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:51:47.365224 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359777 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:51:47.365699 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359780 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:51:47.365699 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359783 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:51:47.365699 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359786 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:51:47.365699 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359789 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:51:47.365699 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359791 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:51:47.365699 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359794 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:51:47.365699 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359796 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:51:47.365699 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359799 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:51:47.365699 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359801 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:51:47.365699 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359804 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:51:47.365699 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359806 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:51:47.365699 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359809 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:51:47.365699 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359811 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:51:47.365699 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359814 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:51:47.365699 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359816 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:51:47.365699 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359818 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:51:47.365699 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359821 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:51:47.365699 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359823 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:51:47.365699 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359826 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:51:47.365699 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359828 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:51:47.366211 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359831 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:51:47.366211 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359834 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:51:47.366211 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359836 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:51:47.366211 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359839 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:51:47.366211 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359841 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:51:47.366211 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359844 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:51:47.366211 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359846 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:51:47.366211 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359849 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:51:47.366211 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359853 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:51:47.366211 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359856 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:51:47.366211 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359876 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:51:47.366211 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359880 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:51:47.366211 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359883 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:51:47.366211 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359886 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:51:47.366211 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359890 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:51:47.366211 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359892 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:51:47.366211 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359895 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:51:47.366211 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359898 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:51:47.366211 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359900 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:51:47.366653 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359903 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:51:47.366653 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359905 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:51:47.366653 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359908 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:51:47.366653 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359911 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:51:47.366653 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359913 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:51:47.366653 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359916 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:51:47.366653 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359918 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:51:47.366653 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359921 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:51:47.366653 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359923 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:51:47.366653 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359926 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:51:47.366653 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359928 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:51:47.366653 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359931 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:51:47.366653 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359933 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:51:47.366653 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359935 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:51:47.366653 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359938 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:51:47.366653 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359940 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:51:47.366653 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359943 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:51:47.366653 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359945 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:51:47.366653 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359948 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:51:47.366653 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359950 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:51:47.367188 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359953 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:51:47.367188 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359956 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:51:47.367188 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359960 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:51:47.367188 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359963 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:51:47.367188 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359966 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:51:47.367188 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359969 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:51:47.367188 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359972 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:51:47.367188 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359975 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:51:47.367188 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359978 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:51:47.367188 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359980 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:51:47.367188 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359983 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:51:47.367188 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359985 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:51:47.367188 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359988 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:51:47.367188 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359990 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:51:47.367188 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359993 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:51:47.367188 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359995 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:51:47.367188 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.359998 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:51:47.367188 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360000 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:51:47.367188 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360003 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:51:47.367649 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.360030 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:51:47.367649 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360160 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:51:47.367649 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360165 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:51:47.367649 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360168 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:51:47.367649 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360171 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:51:47.367649 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360173 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:51:47.367649 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360176 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:51:47.367649 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360179 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:51:47.367649 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360181 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:51:47.367649 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360184 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:51:47.367649 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360187 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:51:47.367649 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360189 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:51:47.367649 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360192 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:51:47.367649 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360194 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:51:47.367649 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360197 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:51:47.367649 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360199 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:51:47.368220 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360201 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:51:47.368220 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360204 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:51:47.368220 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360207 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:51:47.368220 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360210 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:51:47.368220 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360212 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:51:47.368220 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360215 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:51:47.368220 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360218 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:51:47.368220 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360221 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:51:47.368220 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360223 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:51:47.368220 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360226 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:51:47.368220 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360228 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:51:47.368220 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360231 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:51:47.368220 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360234 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:51:47.368220 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360236 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:51:47.368220 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360239 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:51:47.368220 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360241 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:51:47.368220 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360244 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:51:47.368220 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360246 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:51:47.368220 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360248 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:51:47.368673 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360251 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:51:47.368673 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360253 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:51:47.368673 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360256 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:51:47.368673 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360258 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:51:47.368673 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360261 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:51:47.368673 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360263 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:51:47.368673 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360266 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:51:47.368673 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360269 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:51:47.368673 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360271 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:51:47.368673 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360274 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:51:47.368673 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360276 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:51:47.368673 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360279 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:51:47.368673 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360281 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:51:47.368673 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360284 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:51:47.368673 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360286 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:51:47.368673 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360289 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:51:47.368673 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360292 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:51:47.368673 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360294 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:51:47.368673 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360297 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:51:47.369323 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360299 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:51:47.369323 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360302 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:51:47.369323 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360305 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:51:47.369323 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360310 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:51:47.369323 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360312 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:51:47.369323 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360315 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:51:47.369323 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360317 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:51:47.369323 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360321 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:51:47.369323 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360325 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:51:47.369323 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360328 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:51:47.369323 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360331 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:51:47.369323 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360333 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:51:47.369323 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360337 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:51:47.369323 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360342 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:51:47.369323 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360345 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:51:47.369323 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360347 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:51:47.369323 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360350 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:51:47.369323 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360353 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:51:47.369323 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360355 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:51:47.370107 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360358 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:51:47.370107 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360360 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:51:47.370107 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360363 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:51:47.370107 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360366 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:51:47.370107 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360369 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:51:47.370107 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360371 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:51:47.370107 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360373 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:51:47.370107 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360376 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:51:47.370107 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360378 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:51:47.370107 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360381 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:51:47.370107 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360384 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:51:47.370107 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360386 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:51:47.370107 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360389 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:51:47.370107 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360391 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:51:47.370107 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360394 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:51:47.370107 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360397 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:51:47.370107 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360401 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:51:47.370107 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360403 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:51:47.370107 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360406 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:51:47.370107 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360408 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:51:47.370609 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360411 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:51:47.370609 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360414 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:51:47.370609 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360416 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:51:47.370609 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360419 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:51:47.370609 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360421 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:51:47.370609 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360424 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:51:47.370609 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360426 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:51:47.370609 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360428 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:51:47.370609 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360431 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:51:47.370609 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360433 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:51:47.370609 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360436 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:51:47.370609 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360438 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:51:47.370609 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360441 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:51:47.370609 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360443 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:51:47.370609 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.360473 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:51:47.370609 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360503 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:51:47.371070 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360507 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:51:47.371070 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360510 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:51:47.371070 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360513 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:51:47.371070 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360515 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:51:47.371070 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360517 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:51:47.371070 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360520 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:51:47.371070 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360523 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:51:47.371070 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360525 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:51:47.371070 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360528 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:51:47.371070 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360530 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:51:47.371070 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360533 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:51:47.371070 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360535 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:51:47.371070 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360541 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:51:47.371070 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360543 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:51:47.371070 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360546 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:51:47.371070 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360548 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:51:47.371070 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360551 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:51:47.371070 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360553 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:51:47.371070 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360556 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:51:47.371070 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360558 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:51:47.371567 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360561 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:51:47.371567 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360563 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:51:47.371567 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360566 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:51:47.371567 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360568 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:51:47.371567 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360571 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:51:47.371567 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360573 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:51:47.371567 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360576 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:51:47.371567 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360578 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:51:47.371567 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360581 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:51:47.371567 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360583 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:51:47.371567 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360586 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:51:47.371567 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360590 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:51:47.371567 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360593 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:51:47.371567 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360595 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:51:47.371567 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360598 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:51:47.371567 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360600 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:51:47.371567 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360603 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:51:47.371567 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360605 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:51:47.371567 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360608 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:51:47.371567 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360610 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:51:47.372208 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360613 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:51:47.372208 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360615 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:51:47.372208 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360618 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:51:47.372208 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360620 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:51:47.372208 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360624 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:51:47.372208 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360627 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:51:47.372208 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360630 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:51:47.372208 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360633 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:51:47.372208 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360635 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:51:47.372208 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360638 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:51:47.372208 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360640 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:51:47.372208 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360643 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:51:47.372208 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360662 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:51:47.372208 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360667 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:51:47.372208 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360670 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:51:47.372208 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360673 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:51:47.372208 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360675 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:51:47.372208 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360678 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:51:47.372208 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360680 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:51:47.372208 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360683 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:51:47.372788 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360686 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:51:47.372788 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360689 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:51:47.372788 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360692 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:51:47.372788 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360696 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:51:47.372788 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360699 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:51:47.372788 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360702 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:51:47.372788 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360704 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:51:47.372788 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360706 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:51:47.372788 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360709 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:51:47.372788 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360712 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:51:47.372788 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360715 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:51:47.372788 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360717 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:51:47.372788 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360720 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:51:47.372788 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360722 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:51:47.372788 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360725 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:51:47.372788 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360728 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:51:47.372788 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360732 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:51:47.372788 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360734 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:51:47.372788 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360737 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:51:47.373336 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360740 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:51:47.373336 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360743 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:51:47.373336 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360745 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:51:47.373336 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360748 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:51:47.373336 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360751 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:51:47.373336 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360753 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:51:47.373336 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360756 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:51:47.373336 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360759 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:51:47.373336 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360762 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:51:47.373336 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360764 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:51:47.373336 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360767 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:51:47.373336 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360769 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:51:47.373336 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360772 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:51:47.373336 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360774 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:51:47.373336 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360777 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:51:47.373336 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360779 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:51:47.373336 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360783 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:51:47.373336 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360786 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:51:47.373336 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360790 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:51:47.373813 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360793 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:51:47.373813 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360796 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:51:47.373813 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360799 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:51:47.373813 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360802 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:51:47.373813 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360805 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:51:47.373813 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360811 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:51:47.373813 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:47.360814 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:51:47.373813 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.360882 2575 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 22:51:47.373813 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.366631 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 22:51:47.373813 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.367709 2575 server.go:1014] "Starting client certificate rotation" Nov 02 22:51:47.373813 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.367851 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 22:51:47.373813 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.367905 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 22:51:47.373813 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.369908 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 22:51:47.373813 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.371857 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 22:51:47.382786 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.382769 2575 log.go:25] "Validated CRI v1 runtime API" Nov 02 22:51:47.386565 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.386552 2575 log.go:25] "Validated CRI v1 image API" Nov 02 22:51:47.387358 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.387334 2575 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 22:51:47.390023 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.390004 2575 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ab63f09b-d9ce-4d89-a1e7-44b7db89c253:/dev/nvme0n1p3 b143f1c5-992e-460f-80d5-80a31569b717:/dev/nvme0n1p4] Nov 02 22:51:47.390084 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.390023 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 22:51:47.396905 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.396772 2575 manager.go:217] Machine: {Timestamp:2026-11-02 22:51:47.395304958 +0000 UTC m=+0.344909631 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100297 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec227d0787f60e8b090322d0aa81776b SystemUUID:ec227d07-87f6-0e8b-0903-22d0aa81776b BootID:5a4a1bc2-bce4-4bfd-961e-8ec63843ac65 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:e9:c6:37:91 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:e9:c6:37:91 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:1b:6d:74:97:91 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 22:51:47.396905 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.396905 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 22:51:47.397003 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.396985 2575 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 22:51:47.400684 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.400658 2575 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 22:51:47.400821 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.400686 2575 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-211.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 22:51:47.400875 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.400823 2575 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 22:51:47.400875 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.400830 2575 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 22:51:47.400934 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.400881 2575 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 22:51:47.400934 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.400904 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 22:51:47.402202 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.402192 2575 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 22:51:47.406696 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.406684 2575 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 22:51:47.406815 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.406806 2575 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 22:51:47.407668 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.407658 2575 kubelet.go:493] "Attempting to sync node with API server" Nov 02 22:51:47.407704 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.407671 2575 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 22:51:47.407704 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.407681 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 22:51:47.407704 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.407690 2575 kubelet.go:405] "Adding apiserver pod source" Nov 02 22:51:47.407704 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.407699 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 22:51:47.408240 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.408214 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 22:51:47.411797 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.411144 2575 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 22:51:47.418565 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.418549 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 22:51:47.419529 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.419515 2575 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 22:51:47.419569 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.419547 2575 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 22:51:47.419709 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.419697 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 22:51:47.419743 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.419715 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 22:51:47.419743 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.419722 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 22:51:47.419743 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.419727 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 22:51:47.419743 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.419733 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 22:51:47.419743 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.419738 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 22:51:47.420588 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.420579 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 22:51:47.420588 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.420588 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 22:51:47.420648 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.420597 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 22:51:47.420648 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.420604 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 22:51:47.420648 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.420612 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 22:51:47.420648 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.420621 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 22:51:47.422161 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.422151 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 22:51:47.422228 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.422163 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 22:51:47.426255 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.426242 2575 server.go:1263] "Started kubelet" Nov 02 22:51:47.426449 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.426367 2575 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 22:51:47.426502 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.426375 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 22:51:47.426502 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.426479 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 22:51:47.427017 ip-10-0-130-211 systemd[1]: Started Kubernetes Kubelet. Nov 02 22:51:47.430289 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.429830 2575 server.go:317] "Adding debug handlers to kubelet server" Nov 02 22:51:47.430289 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.429986 2575 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 22:51:47.432857 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:47.432838 2575 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 22:51:47.435750 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.435730 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 22:51:47.437036 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.437008 2575 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 22:51:47.437137 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:47.437114 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-211.us-west-2.compute.internal\" not found" Nov 02 22:51:47.437804 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.437786 2575 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 22:51:47.437804 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.437806 2575 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 22:51:47.437957 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.437784 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 22:51:47.437957 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.437901 2575 reconstruct.go:97] "Volume reconstruction finished" Nov 02 22:51:47.437957 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.437910 2575 reconciler.go:29] "Reconciler: start to sync state" Nov 02 22:51:47.437957 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.437910 2575 factory.go:55] Registering systemd factory Nov 02 22:51:47.437957 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.437928 2575 factory.go:223] Registration of the systemd container factory successfully Nov 02 22:51:47.438201 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.438188 2575 factory.go:153] Registering CRI-O factory Nov 02 22:51:47.438241 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.438203 2575 factory.go:223] Registration of the crio container factory successfully Nov 02 22:51:47.438271 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.438245 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 22:51:47.438271 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.438260 2575 factory.go:103] Registering Raw factory Nov 02 22:51:47.438271 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.438270 2575 manager.go:1199] Started watching for new ooms in manager Nov 02 22:51:47.439355 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.439335 2575 manager.go:319] Starting recovery of all containers Nov 02 22:51:47.449499 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.449474 2575 manager.go:324] Recovery completed Nov 02 22:51:47.454895 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.454879 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:51:47.456687 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.456658 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-211.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:51:47.456773 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.456699 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-211.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:51:47.456773 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.456714 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-211.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:51:47.457282 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.457267 2575 cpu_manager.go:226] "Starting" policy="none" Nov 02 22:51:47.457282 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.457278 2575 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 22:51:47.457394 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.457299 2575 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 22:51:47.457394 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.457304 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:51:47.458026 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.458006 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-211.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:51:47.458092 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.458040 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-211.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:51:47.458092 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.458056 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-211.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:51:47.459495 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.459470 2575 policy_none.go:50] "Start" Nov 02 22:51:47.459495 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.459489 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 22:51:47.459495 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.459499 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 22:51:47.461414 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.461401 2575 policy_none.go:44] "Start" Nov 02 22:51:47.510534 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.510487 2575 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 22:51:47.514778 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.514760 2575 manager.go:341] "Starting Device Plugin manager" Nov 02 22:51:47.514912 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:47.514798 2575 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 22:51:47.514912 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.514811 2575 server.go:87] "Starting device plugin registration server" Nov 02 22:51:47.515158 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.515142 2575 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 22:51:47.515216 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.515159 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 22:51:47.515319 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.515302 2575 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 22:51:47.515422 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.515405 2575 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 22:51:47.515422 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.515418 2575 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 22:51:47.516012 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:47.515991 2575 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 22:51:47.516100 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:47.516055 2575 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-211.us-west-2.compute.internal\" not found" Nov 02 22:51:47.554718 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.554691 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-211.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 22:51:47.556247 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:47.554702 2575 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-211.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 22:51:47.574229 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.574198 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 22:51:47.575444 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.575422 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 22:51:47.575444 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.575444 2575 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 22:51:47.575591 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.575465 2575 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 22:51:47.575591 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:47.575496 2575 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 22:51:47.601447 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:47.600470 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-211.us-west-2.compute.internal.18e45c2ebdffe4d4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-211.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-211.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 22:51:47.426215124 +0000 UTC m=+0.375819797,LastTimestamp:2026-11-02 22:51:47.426215124 +0000 UTC m=+0.375819797,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-211.us-west-2.compute.internal,}" Nov 02 22:51:47.615232 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:47.615207 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 02 22:51:47.615232 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.615220 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:51:47.616077 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.616060 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-211.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:51:47.616161 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.616089 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-211.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:51:47.616161 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.616099 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-211.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:51:47.616161 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.616123 2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:51:47.622407 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.622389 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 22:51:47.622481 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.622432 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sf4t7" Nov 02 22:51:47.676187 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.676008 2575 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-211.us-west-2.compute.internal"] Nov 02 22:51:47.676291 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.676222 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:51:47.677354 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.677337 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-211.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:51:47.677416 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.677368 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-211.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:51:47.677416 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.677379 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-211.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:51:47.679370 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.679358 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:51:47.679963 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.679939 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:51:47.680061 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.679979 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:51:47.680123 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.680091 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-211.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:51:47.680123 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.680123 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-211.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:51:47.680200 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.680138 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-211.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:51:47.680660 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.680645 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-211.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:51:47.680731 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.680674 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-211.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:51:47.680731 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.680690 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-211.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:51:47.681944 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.681929 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:51:47.682012 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.681955 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:51:47.682641 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.682627 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-211.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:51:47.682701 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.682656 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-211.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:51:47.682701 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.682667 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-211.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:51:47.715223 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:47.715199 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-211.us-west-2.compute.internal\" not found" node="ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:51:47.718525 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:47.718505 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-211.us-west-2.compute.internal\" not found" node="ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:51:47.720164 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.720148 2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:51:47.720887 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.720873 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:51:47.721371 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.721357 2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:51:47.739699 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.739673 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f2cabf9747a94425892740afb80ec0fc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal\" (UID: \"f2cabf9747a94425892740afb80ec0fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:51:47.739782 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.739703 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2cabf9747a94425892740afb80ec0fc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal\" (UID: \"f2cabf9747a94425892740afb80ec0fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:51:47.739782 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.739720 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7ba3fed7e8adcc2e1f404604f948f337-config\") pod \"kube-apiserver-proxy-ip-10-0-130-211.us-west-2.compute.internal\" (UID: \"7ba3fed7e8adcc2e1f404604f948f337\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:51:47.773846 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.773818 2575 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:51:47.773846 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:47.773847 2575 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-211.us-west-2.compute.internal\": node \"ip-10-0-130-211.us-west-2.compute.internal\" not found" Nov 02 22:51:47.837885 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.837851 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:51:47.840051 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.840031 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f2cabf9747a94425892740afb80ec0fc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal\" (UID: \"f2cabf9747a94425892740afb80ec0fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:51:47.840101 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.840060 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2cabf9747a94425892740afb80ec0fc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal\" (UID: \"f2cabf9747a94425892740afb80ec0fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:51:47.840101 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.840081 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7ba3fed7e8adcc2e1f404604f948f337-config\") pod \"kube-apiserver-proxy-ip-10-0-130-211.us-west-2.compute.internal\" (UID: \"7ba3fed7e8adcc2e1f404604f948f337\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:51:47.840173 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.840119 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7ba3fed7e8adcc2e1f404604f948f337-config\") pod \"kube-apiserver-proxy-ip-10-0-130-211.us-west-2.compute.internal\" (UID: \"7ba3fed7e8adcc2e1f404604f948f337\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:51:47.840173 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.840133 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2cabf9747a94425892740afb80ec0fc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal\" (UID: \"f2cabf9747a94425892740afb80ec0fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:51:47.840173 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.840119 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f2cabf9747a94425892740afb80ec0fc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal\" (UID: \"f2cabf9747a94425892740afb80ec0fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:51:47.938854 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.938781 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:51:47.939928 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:47.939914 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:51:48.018080 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:48.018048 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:51:48.021475 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:48.021458 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:51:48.040828 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:48.040807 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:51:49.642704 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:49.642676 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 22:51:49.643227 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:49.642871 2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 22:51:49.643227 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:49.642870 2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 22:51:49.643227 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:49.642903 2575 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a2b73d5d16d574ed99182329d58f278a-a10e0d37d7b270df.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-130-211.us-west-2.compute.internal.18e45c2ebfd0d981\": read tcp 10.0.130.211:48658->52.14.154.205:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-130-211.us-west-2.compute.internal.18e45c2ebfd0d981 default 4857 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-211.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-211.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-211.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 22:51:47 +0000 UTC,LastTimestamp:2026-11-02 22:51:47.677354895 +0000 UTC m=+0.626959568,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-211.us-west-2.compute.internal,}" Nov 02 22:51:49.683721 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:49.683695 2575 apiserver.go:52] "Watching apiserver" Nov 02 22:51:49.711560 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:49.711528 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 22:51:49.901015 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:49.900925 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 22:46:48 +0000 UTC" deadline="2028-08-08 02:25:14.020515728 +0000 UTC" Nov 02 22:51:49.901015 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:49.900975 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15459h33m24.119558848s" Nov 02 22:51:49.957906 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:49.957867 2575 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 22:51:49.958187 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:49.958169 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-k4gfs","openshift-cluster-node-tuning-operator/tuned-zjn7n","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-211.us-west-2.compute.internal","openshift-network-operator/iptables-alerter-92hm7","openshift-ovn-kubernetes/ovnkube-node-bp6bf","kube-system/konnectivity-agent-zqssw","openshift-dns/node-resolver-nrxwb","openshift-image-registry/node-ca-4mhgx","openshift-multus/multus-njn94","openshift-network-diagnostics/network-check-target-rmgpt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv","openshift-multus/multus-additional-cni-plugins-cs2mc","openshift-multus/network-metrics-daemon-9zqrz"] Nov 02 22:51:49.960611 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:49.960592 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4mhgx" Nov 02 22:51:49.962555 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:49.962527 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:51:49.964559 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:49.964542 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zqssw" Nov 02 22:51:49.966516 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:49.966501 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-92hm7" Nov 02 22:51:49.968582 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:49.968566 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:49.970720 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:49.970700 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:51:49.970801 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:49.970772 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4gfs" podUID="fd96552a-1a78-4fac-8faf-71d36f010905" Nov 02 22:51:49.972673 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:49.972655 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nrxwb" Nov 02 22:51:49.974683 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:49.974668 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-njn94" Nov 02 22:51:49.976754 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:49.976740 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rmgpt" Nov 02 22:51:49.976842 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:49.976807 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rmgpt" podUID="e3a3bc57-6813-441c-bdc0-d39eb8214c61" Nov 02 22:51:49.978908 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:49.978882 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:49.981132 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:49.981117 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:51:49.983262 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:49.983246 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zqrz" Nov 02 22:51:49.983330 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:49.983303 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zqrz" podUID="fc69f07f-4001-4951-aff8-d3d6671cc5d4" Nov 02 22:51:50.014394 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.014363 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 22:51:50.014532 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.014515 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 22:51:50.014592 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.014543 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 22:51:50.014649 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.014589 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 22:51:50.014783 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.014768 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9qgnp\"" Nov 02 22:51:50.015554 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.015536 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 22:51:50.015643 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.015629 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 22:51:50.015791 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.015777 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sdz5j\"" Nov 02 22:51:50.015882 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.015867 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 22:51:50.017717 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.017616 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 22:51:50.017717 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.017696 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-crtkq\"" Nov 02 22:51:50.017802 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.017746 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 22:51:50.019437 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.019415 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-host-run-multus-certs\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.019533 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.019448 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cb1743a1-6e7b-4f99-80e6-197a6916580d-registration-dir\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:51:50.019533 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.019502 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctsxp\" (UniqueName: \"kubernetes.io/projected/4f8b41ab-cbec-4f72-a5d9-853953748c0b-kube-api-access-ctsxp\") pod \"iptables-alerter-92hm7\" (UID: \"4f8b41ab-cbec-4f72-a5d9-853953748c0b\") " pod="openshift-network-operator/iptables-alerter-92hm7" Nov 02 22:51:50.019659 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.019540 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:51:50.019659 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.019546 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 22:51:50.019659 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.019548 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cb1743a1-6e7b-4f99-80e6-197a6916580d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:51:50.019659 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.019592 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-77brg\"" Nov 02 22:51:50.019809 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.019596 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cb1743a1-6e7b-4f99-80e6-197a6916580d-device-dir\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:51:50.019861 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.019832 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rt8s\" (UniqueName: \"kubernetes.io/projected/cb1743a1-6e7b-4f99-80e6-197a6916580d-kube-api-access-6rt8s\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:51:50.019922 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.019870 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8fbcc658-b377-4da4-bf1d-e6274dfd805b-konnectivity-ca\") pod \"konnectivity-agent-zqssw\" (UID: \"8fbcc658-b377-4da4-bf1d-e6274dfd805b\") " pod="kube-system/konnectivity-agent-zqssw" Nov 02 22:51:50.019922 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.019915 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-slash\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.020017 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.019950 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-run-openvswitch\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.020017 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.019975 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 22:51:50.020017 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.019980 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-ovnkube-config\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.020017 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.020007 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-lib-modules\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.020229 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.020058 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-node-log\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.020229 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.020104 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-cni-bin\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.020229 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.020158 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-etc-kubernetes\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.020229 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.020214 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-var-lib-kubelet\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.020406 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.020263 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-var-lib-openvswitch\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.020531 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.020496 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-host-run-netns\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.020587 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.020534 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-etc-systemd\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.020587 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.020558 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzxgm\" (UniqueName: \"kubernetes.io/projected/6b3753c7-7634-4209-8760-7a52da534549-kube-api-access-wzxgm\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.020676 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.020589 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cb1743a1-6e7b-4f99-80e6-197a6916580d-etc-selinux\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:51:50.020676 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.020617 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-systemd-units\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.020676 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.020646 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-etc-openvswitch\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.020676 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.020671 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-log-socket\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.020851 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.020699 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d3575a78-aad1-41e0-bad4-63724365aeb1-hosts-file\") pod \"node-resolver-nrxwb\" (UID: \"d3575a78-aad1-41e0-bad4-63724365aeb1\") " pod="openshift-dns/node-resolver-nrxwb" Nov 02 22:51:50.020851 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.020728 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7qr5\" (UniqueName: \"kubernetes.io/projected/ee52c167-6361-4adb-9899-93ca4c2128b8-kube-api-access-q7qr5\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.020851 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.020756 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-etc-sysctl-conf\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.020851 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.020785 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f8b41ab-cbec-4f72-a5d9-853953748c0b-iptables-alerter-script\") pod \"iptables-alerter-92hm7\" (UID: \"4f8b41ab-cbec-4f72-a5d9-853953748c0b\") " pod="openshift-network-operator/iptables-alerter-92hm7" Nov 02 22:51:50.020851 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.020818 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-ovnkube-script-lib\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.020851 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.020847 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-multus-cni-dir\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.021102 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.021032 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7nrp\" (UniqueName: \"kubernetes.io/projected/e3a3bc57-6813-441c-bdc0-d39eb8214c61-kube-api-access-q7nrp\") pod \"network-check-target-rmgpt\" (UID: \"e3a3bc57-6813-441c-bdc0-d39eb8214c61\") " pod="openshift-network-diagnostics/network-check-target-rmgpt" Nov 02 22:51:50.021102 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.021069 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-etc-kubernetes\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.021226 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.021104 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/668094b0-0e10-47aa-ba49-6944f7083bc6-system-cni-dir\") pod \"multus-additional-cni-plugins-cs2mc\" (UID: \"668094b0-0e10-47aa-ba49-6944f7083bc6\") " pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:51:50.021226 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.021149 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8fbcc658-b377-4da4-bf1d-e6274dfd805b-agent-certs\") pod \"konnectivity-agent-zqssw\" (UID: \"8fbcc658-b377-4da4-bf1d-e6274dfd805b\") " pod="kube-system/konnectivity-agent-zqssw" Nov 02 22:51:50.021226 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.021173 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-ovn-node-metrics-cert\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.021226 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.021204 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-system-cni-dir\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.021382 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.021236 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-host-var-lib-kubelet\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.021382 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.021301 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/668094b0-0e10-47aa-ba49-6944f7083bc6-cni-binary-copy\") pod \"multus-additional-cni-plugins-cs2mc\" (UID: \"668094b0-0e10-47aa-ba49-6944f7083bc6\") " pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:51:50.021382 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.021329 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/668094b0-0e10-47aa-ba49-6944f7083bc6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cs2mc\" (UID: \"668094b0-0e10-47aa-ba49-6944f7083bc6\") " pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:51:50.021382 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.021368 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-host-run-k8s-cni-cncf-io\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.021538 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.021401 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4shq\" (UniqueName: \"kubernetes.io/projected/668094b0-0e10-47aa-ba49-6944f7083bc6-kube-api-access-k4shq\") pod \"multus-additional-cni-plugins-cs2mc\" (UID: \"668094b0-0e10-47aa-ba49-6944f7083bc6\") " pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:51:50.021538 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.021434 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-kubelet\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.021538 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.021465 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fd96552a-1a78-4fac-8faf-71d36f010905-kubelet-config\") pod \"global-pull-secret-syncer-k4gfs\" (UID: \"fd96552a-1a78-4fac-8faf-71d36f010905\") " pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:51:50.021538 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.021497 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee52c167-6361-4adb-9899-93ca4c2128b8-cni-binary-copy\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.021538 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.021522 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b3753c7-7634-4209-8760-7a52da534549-tmp\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.021732 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.021609 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/668094b0-0e10-47aa-ba49-6944f7083bc6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cs2mc\" (UID: \"668094b0-0e10-47aa-ba49-6944f7083bc6\") " pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:51:50.021732 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.021638 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-run-systemd\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.021732 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.021683 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkqkc\" (UniqueName: \"kubernetes.io/projected/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-kube-api-access-kkqkc\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.021732 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.021715 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-multus-socket-dir-parent\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.021884 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.021744 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a464bf3-77fa-4fcd-8593-327fa1f4a19a-host\") pod \"node-ca-4mhgx\" (UID: \"3a464bf3-77fa-4fcd-8593-327fa1f4a19a\") " pod="openshift-image-registry/node-ca-4mhgx" Nov 02 22:51:50.021884 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.021776 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cb1743a1-6e7b-4f99-80e6-197a6916580d-socket-dir\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:51:50.021884 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.021808 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-run-netns\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.021884 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.021838 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-run-ovn\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.022061 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.021874 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-host-var-lib-cni-multus\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.022061 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.021928 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-etc-sysctl-d\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.022061 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.021958 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-run-ovn-kubernetes\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.022061 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.022037 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-sys\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.022219 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.022091 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/668094b0-0e10-47aa-ba49-6944f7083bc6-cnibin\") pod \"multus-additional-cni-plugins-cs2mc\" (UID: \"668094b0-0e10-47aa-ba49-6944f7083bc6\") " pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:51:50.022219 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.022170 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/668094b0-0e10-47aa-ba49-6944f7083bc6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cs2mc\" (UID: \"668094b0-0e10-47aa-ba49-6944f7083bc6\") " pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:51:50.022301 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.022243 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-env-overrides\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.022301 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.022267 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6jlv\" (UniqueName: \"kubernetes.io/projected/fc69f07f-4001-4951-aff8-d3d6671cc5d4-kube-api-access-n6jlv\") pod \"network-metrics-daemon-9zqrz\" (UID: \"fc69f07f-4001-4951-aff8-d3d6671cc5d4\") " pod="openshift-multus/network-metrics-daemon-9zqrz" Nov 02 22:51:50.022301 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.022285 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fd96552a-1a78-4fac-8faf-71d36f010905-dbus\") pod \"global-pull-secret-syncer-k4gfs\" (UID: \"fd96552a-1a78-4fac-8faf-71d36f010905\") " pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:51:50.022421 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.022301 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvg2k\" (UniqueName: \"kubernetes.io/projected/d3575a78-aad1-41e0-bad4-63724365aeb1-kube-api-access-vvg2k\") pod \"node-resolver-nrxwb\" (UID: \"d3575a78-aad1-41e0-bad4-63724365aeb1\") " pod="openshift-dns/node-resolver-nrxwb" Nov 02 22:51:50.022421 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.022322 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-etc-modprobe-d\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.022421 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.022414 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-etc-sysconfig\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.022542 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.022432 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-run\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.022542 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.022452 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3575a78-aad1-41e0-bad4-63724365aeb1-tmp-dir\") pod \"node-resolver-nrxwb\" (UID: \"d3575a78-aad1-41e0-bad4-63724365aeb1\") " pod="openshift-dns/node-resolver-nrxwb" Nov 02 22:51:50.022542 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.022476 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-hostroot\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.022542 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.022501 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-multus-conf-dir\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.022542 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.022534 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cml47\" (UniqueName: \"kubernetes.io/projected/3a464bf3-77fa-4fcd-8593-327fa1f4a19a-kube-api-access-cml47\") pod \"node-ca-4mhgx\" (UID: \"3a464bf3-77fa-4fcd-8593-327fa1f4a19a\") " pod="openshift-image-registry/node-ca-4mhgx" Nov 02 22:51:50.022791 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.022565 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-host\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.022791 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.022586 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6b3753c7-7634-4209-8760-7a52da534549-etc-tuned\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.022791 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.022618 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ee52c167-6361-4adb-9899-93ca4c2128b8-multus-daemon-config\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.022791 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.022741 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3a464bf3-77fa-4fcd-8593-327fa1f4a19a-serviceca\") pod \"node-ca-4mhgx\" (UID: \"3a464bf3-77fa-4fcd-8593-327fa1f4a19a\") " pod="openshift-image-registry/node-ca-4mhgx" Nov 02 22:51:50.022791 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.022776 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs\") pod \"network-metrics-daemon-9zqrz\" (UID: \"fc69f07f-4001-4951-aff8-d3d6671cc5d4\") " pod="openshift-multus/network-metrics-daemon-9zqrz" Nov 02 22:51:50.023010 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.022803 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cb1743a1-6e7b-4f99-80e6-197a6916580d-sys-fs\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:51:50.023010 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.022820 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f8b41ab-cbec-4f72-a5d9-853953748c0b-host-slash\") pod \"iptables-alerter-92hm7\" (UID: \"4f8b41ab-cbec-4f72-a5d9-853953748c0b\") " pod="openshift-network-operator/iptables-alerter-92hm7" Nov 02 22:51:50.023010 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.022871 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-cni-netd\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.023010 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.022923 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.023010 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.022982 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd96552a-1a78-4fac-8faf-71d36f010905-original-pull-secret\") pod \"global-pull-secret-syncer-k4gfs\" (UID: \"fd96552a-1a78-4fac-8faf-71d36f010905\") " pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:51:50.023207 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.023020 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-cnibin\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.023207 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.023073 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-host-var-lib-cni-bin\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.062859 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.062831 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 22:51:50.062859 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.062833 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 22:51:50.063095 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.062872 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-c9lzr\"" Nov 02 22:51:50.111412 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.111382 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 22:51:50.111521 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.111440 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-knnhn" Nov 02 22:51:50.115393 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.115372 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 22:51:50.115518 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.115510 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 22:51:50.115576 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.115525 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 22:51:50.115576 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.115522 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 22:51:50.115676 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.115645 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rxqrv\"" Nov 02 22:51:50.115676 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.115652 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 22:51:50.115676 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.115665 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:51:50.115816 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.115687 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 22:51:50.115816 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.115692 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-92nqq\"" Nov 02 22:51:50.115816 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.115705 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9p98l\"" Nov 02 22:51:50.115816 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.115711 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 22:51:50.115816 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.115693 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 22:51:50.115816 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.115698 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 22:51:50.115816 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.115718 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 22:51:50.115816 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.115739 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 22:51:50.115816 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.115755 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-v5hvl\"" Nov 02 22:51:50.115816 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.115760 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 22:51:50.115816 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.115796 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 22:51:50.123432 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.123415 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cb1743a1-6e7b-4f99-80e6-197a6916580d-socket-dir\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:51:50.123495 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.123441 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-run-netns\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.123495 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.123456 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-run-ovn\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.123495 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.123478 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-host-var-lib-cni-multus\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.123639 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.123508 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-etc-sysctl-d\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.123639 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.123544 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-run-netns\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.123639 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.123562 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-host-var-lib-cni-multus\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.123639 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.123585 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cb1743a1-6e7b-4f99-80e6-197a6916580d-socket-dir\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:51:50.123639 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.123589 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-run-ovn\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.123639 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.123603 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-run-ovn-kubernetes\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.123639 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.123634 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-sys\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.123639 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.123639 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-etc-sysctl-d\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.124024 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.123659 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/668094b0-0e10-47aa-ba49-6944f7083bc6-cnibin\") pod \"multus-additional-cni-plugins-cs2mc\" (UID: \"668094b0-0e10-47aa-ba49-6944f7083bc6\") " pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:51:50.124024 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.123670 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-run-ovn-kubernetes\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.124024 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.123685 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-sys\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.124024 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.123690 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/668094b0-0e10-47aa-ba49-6944f7083bc6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cs2mc\" (UID: \"668094b0-0e10-47aa-ba49-6944f7083bc6\") " pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:51:50.124024 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.123688 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/668094b0-0e10-47aa-ba49-6944f7083bc6-cnibin\") pod \"multus-additional-cni-plugins-cs2mc\" (UID: \"668094b0-0e10-47aa-ba49-6944f7083bc6\") " pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:51:50.124024 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.123714 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-env-overrides\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.124024 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.123740 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6jlv\" (UniqueName: \"kubernetes.io/projected/fc69f07f-4001-4951-aff8-d3d6671cc5d4-kube-api-access-n6jlv\") pod \"network-metrics-daemon-9zqrz\" (UID: \"fc69f07f-4001-4951-aff8-d3d6671cc5d4\") " pod="openshift-multus/network-metrics-daemon-9zqrz" Nov 02 22:51:50.124024 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.123765 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fd96552a-1a78-4fac-8faf-71d36f010905-dbus\") pod \"global-pull-secret-syncer-k4gfs\" (UID: \"fd96552a-1a78-4fac-8faf-71d36f010905\") " pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:51:50.124024 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.123791 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvg2k\" (UniqueName: \"kubernetes.io/projected/d3575a78-aad1-41e0-bad4-63724365aeb1-kube-api-access-vvg2k\") pod \"node-resolver-nrxwb\" (UID: \"d3575a78-aad1-41e0-bad4-63724365aeb1\") " pod="openshift-dns/node-resolver-nrxwb" Nov 02 22:51:50.124024 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.123816 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-etc-modprobe-d\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.124024 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.123838 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-etc-sysconfig\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.124024 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.123861 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-run\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.124024 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.123955 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fd96552a-1a78-4fac-8faf-71d36f010905-dbus\") pod \"global-pull-secret-syncer-k4gfs\" (UID: \"fd96552a-1a78-4fac-8faf-71d36f010905\") " pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:51:50.124024 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.123967 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-etc-modprobe-d\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.124024 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.123973 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-run\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.124024 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124011 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-etc-sysconfig\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.124024 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124016 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3575a78-aad1-41e0-bad4-63724365aeb1-tmp-dir\") pod \"node-resolver-nrxwb\" (UID: \"d3575a78-aad1-41e0-bad4-63724365aeb1\") " pod="openshift-dns/node-resolver-nrxwb" Nov 02 22:51:50.124716 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124047 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-hostroot\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.124716 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124082 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-multus-conf-dir\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.124716 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124086 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-hostroot\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.124716 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124107 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cml47\" (UniqueName: \"kubernetes.io/projected/3a464bf3-77fa-4fcd-8593-327fa1f4a19a-kube-api-access-cml47\") pod \"node-ca-4mhgx\" (UID: \"3a464bf3-77fa-4fcd-8593-327fa1f4a19a\") " pod="openshift-image-registry/node-ca-4mhgx" Nov 02 22:51:50.124716 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124123 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-multus-conf-dir\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.124716 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124131 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-host\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.124716 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124162 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6b3753c7-7634-4209-8760-7a52da534549-etc-tuned\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.124716 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124176 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-host\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.124716 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124189 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ee52c167-6361-4adb-9899-93ca4c2128b8-multus-daemon-config\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.124716 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124212 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3a464bf3-77fa-4fcd-8593-327fa1f4a19a-serviceca\") pod \"node-ca-4mhgx\" (UID: \"3a464bf3-77fa-4fcd-8593-327fa1f4a19a\") " pod="openshift-image-registry/node-ca-4mhgx" Nov 02 22:51:50.124716 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124236 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs\") pod \"network-metrics-daemon-9zqrz\" (UID: \"fc69f07f-4001-4951-aff8-d3d6671cc5d4\") " pod="openshift-multus/network-metrics-daemon-9zqrz" Nov 02 22:51:50.124716 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124236 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-env-overrides\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.124716 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124254 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3575a78-aad1-41e0-bad4-63724365aeb1-tmp-dir\") pod \"node-resolver-nrxwb\" (UID: \"d3575a78-aad1-41e0-bad4-63724365aeb1\") " pod="openshift-dns/node-resolver-nrxwb" Nov 02 22:51:50.124716 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124323 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cb1743a1-6e7b-4f99-80e6-197a6916580d-sys-fs\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:51:50.124716 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:50.124343 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:51:50.124716 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124348 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f8b41ab-cbec-4f72-a5d9-853953748c0b-host-slash\") pod \"iptables-alerter-92hm7\" (UID: \"4f8b41ab-cbec-4f72-a5d9-853953748c0b\") " pod="openshift-network-operator/iptables-alerter-92hm7" Nov 02 22:51:50.124716 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124363 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/668094b0-0e10-47aa-ba49-6944f7083bc6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cs2mc\" (UID: \"668094b0-0e10-47aa-ba49-6944f7083bc6\") " pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:51:50.124716 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124390 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-cni-netd\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.125409 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:50.124401 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs podName:fc69f07f-4001-4951-aff8-d3d6671cc5d4 nodeName:}" failed. No retries permitted until 2026-11-02 22:51:50.624383232 +0000 UTC m=+2.298960028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs") pod "network-metrics-daemon-9zqrz" (UID: "fc69f07f-4001-4951-aff8-d3d6671cc5d4") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:51:50.125409 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124423 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f8b41ab-cbec-4f72-a5d9-853953748c0b-host-slash\") pod \"iptables-alerter-92hm7\" (UID: \"4f8b41ab-cbec-4f72-a5d9-853953748c0b\") " pod="openshift-network-operator/iptables-alerter-92hm7" Nov 02 22:51:50.125409 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124428 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-cni-netd\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.125409 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124440 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.125409 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124430 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cb1743a1-6e7b-4f99-80e6-197a6916580d-sys-fs\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:51:50.125409 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124472 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd96552a-1a78-4fac-8faf-71d36f010905-original-pull-secret\") pod \"global-pull-secret-syncer-k4gfs\" (UID: \"fd96552a-1a78-4fac-8faf-71d36f010905\") " pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:51:50.125409 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124436 2575 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 22:51:50.125409 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124496 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-cnibin\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.125409 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124496 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.125409 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124519 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-host-var-lib-cni-bin\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.125409 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124546 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-host-run-multus-certs\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.125409 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124563 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-cnibin\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.125409 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124570 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cb1743a1-6e7b-4f99-80e6-197a6916580d-registration-dir\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:51:50.125409 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:50.124574 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:51:50.125409 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124607 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-host-var-lib-cni-bin\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.125409 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124613 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctsxp\" (UniqueName: \"kubernetes.io/projected/4f8b41ab-cbec-4f72-a5d9-853953748c0b-kube-api-access-ctsxp\") pod \"iptables-alerter-92hm7\" (UID: \"4f8b41ab-cbec-4f72-a5d9-853953748c0b\") " pod="openshift-network-operator/iptables-alerter-92hm7" Nov 02 22:51:50.125409 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124634 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-host-run-multus-certs\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.126173 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124643 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cb1743a1-6e7b-4f99-80e6-197a6916580d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:51:50.126173 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124657 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cb1743a1-6e7b-4f99-80e6-197a6916580d-registration-dir\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:51:50.126173 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:50.124676 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd96552a-1a78-4fac-8faf-71d36f010905-original-pull-secret podName:fd96552a-1a78-4fac-8faf-71d36f010905 nodeName:}" failed. No retries permitted until 2026-11-02 22:51:50.624658279 +0000 UTC m=+2.299235064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd96552a-1a78-4fac-8faf-71d36f010905-original-pull-secret") pod "global-pull-secret-syncer-k4gfs" (UID: "fd96552a-1a78-4fac-8faf-71d36f010905") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:51:50.126173 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124690 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cb1743a1-6e7b-4f99-80e6-197a6916580d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:51:50.126173 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124675 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3a464bf3-77fa-4fcd-8593-327fa1f4a19a-serviceca\") pod \"node-ca-4mhgx\" (UID: \"3a464bf3-77fa-4fcd-8593-327fa1f4a19a\") " pod="openshift-image-registry/node-ca-4mhgx" Nov 02 22:51:50.126173 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124725 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cb1743a1-6e7b-4f99-80e6-197a6916580d-device-dir\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:51:50.126173 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124731 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ee52c167-6361-4adb-9899-93ca4c2128b8-multus-daemon-config\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.126173 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124752 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rt8s\" (UniqueName: \"kubernetes.io/projected/cb1743a1-6e7b-4f99-80e6-197a6916580d-kube-api-access-6rt8s\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:51:50.126173 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124775 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8fbcc658-b377-4da4-bf1d-e6274dfd805b-konnectivity-ca\") pod \"konnectivity-agent-zqssw\" (UID: \"8fbcc658-b377-4da4-bf1d-e6274dfd805b\") " pod="kube-system/konnectivity-agent-zqssw" Nov 02 22:51:50.126173 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124783 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cb1743a1-6e7b-4f99-80e6-197a6916580d-device-dir\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:51:50.126173 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124790 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-slash\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.126173 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124826 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-slash\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.126173 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124911 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-run-openvswitch\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.126173 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124927 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-run-openvswitch\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.126173 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124950 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-ovnkube-config\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.126173 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.124995 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-lib-modules\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.127091 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125021 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-node-log\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.127091 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125053 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-cni-bin\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.127091 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125082 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-etc-kubernetes\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.127091 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125105 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-var-lib-kubelet\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.127091 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125115 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-lib-modules\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.127091 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125118 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-node-log\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.127091 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125129 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-var-lib-openvswitch\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.127091 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125154 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-host-run-netns\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.127091 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125162 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-etc-kubernetes\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.127091 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125166 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-var-lib-kubelet\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.127091 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125121 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-cni-bin\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.127091 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125177 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-etc-systemd\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.127091 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125201 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzxgm\" (UniqueName: \"kubernetes.io/projected/6b3753c7-7634-4209-8760-7a52da534549-kube-api-access-wzxgm\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.127091 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125202 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-var-lib-openvswitch\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.127091 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125217 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-host-run-netns\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.127091 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125245 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-etc-systemd\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.127091 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125269 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8fbcc658-b377-4da4-bf1d-e6274dfd805b-konnectivity-ca\") pod \"konnectivity-agent-zqssw\" (UID: \"8fbcc658-b377-4da4-bf1d-e6274dfd805b\") " pod="kube-system/konnectivity-agent-zqssw" Nov 02 22:51:50.127091 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125268 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cb1743a1-6e7b-4f99-80e6-197a6916580d-etc-selinux\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:51:50.127976 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125310 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-systemd-units\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.127976 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125325 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cb1743a1-6e7b-4f99-80e6-197a6916580d-etc-selinux\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:51:50.127976 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125329 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-etc-openvswitch\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.127976 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125353 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-log-socket\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.127976 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125355 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-etc-openvswitch\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.127976 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125383 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-systemd-units\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.127976 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125373 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d3575a78-aad1-41e0-bad4-63724365aeb1-hosts-file\") pod \"node-resolver-nrxwb\" (UID: \"d3575a78-aad1-41e0-bad4-63724365aeb1\") " pod="openshift-dns/node-resolver-nrxwb" Nov 02 22:51:50.127976 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125402 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d3575a78-aad1-41e0-bad4-63724365aeb1-hosts-file\") pod \"node-resolver-nrxwb\" (UID: \"d3575a78-aad1-41e0-bad4-63724365aeb1\") " pod="openshift-dns/node-resolver-nrxwb" Nov 02 22:51:50.127976 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125404 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-ovnkube-config\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.127976 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125401 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-log-socket\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.127976 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125418 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7qr5\" (UniqueName: \"kubernetes.io/projected/ee52c167-6361-4adb-9899-93ca4c2128b8-kube-api-access-q7qr5\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.127976 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125435 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-etc-sysctl-conf\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.127976 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125456 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f8b41ab-cbec-4f72-a5d9-853953748c0b-iptables-alerter-script\") pod \"iptables-alerter-92hm7\" (UID: \"4f8b41ab-cbec-4f72-a5d9-853953748c0b\") " pod="openshift-network-operator/iptables-alerter-92hm7" Nov 02 22:51:50.127976 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125472 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-ovnkube-script-lib\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.127976 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125486 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-multus-cni-dir\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.127976 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125503 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7nrp\" (UniqueName: \"kubernetes.io/projected/e3a3bc57-6813-441c-bdc0-d39eb8214c61-kube-api-access-q7nrp\") pod \"network-check-target-rmgpt\" (UID: \"e3a3bc57-6813-441c-bdc0-d39eb8214c61\") " pod="openshift-network-diagnostics/network-check-target-rmgpt" Nov 02 22:51:50.127976 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125526 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-etc-kubernetes\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.128450 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125553 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/668094b0-0e10-47aa-ba49-6944f7083bc6-system-cni-dir\") pod \"multus-additional-cni-plugins-cs2mc\" (UID: \"668094b0-0e10-47aa-ba49-6944f7083bc6\") " pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:51:50.128450 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125565 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-multus-cni-dir\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.128450 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125576 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-etc-sysctl-conf\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.128450 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125599 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8fbcc658-b377-4da4-bf1d-e6274dfd805b-agent-certs\") pod \"konnectivity-agent-zqssw\" (UID: \"8fbcc658-b377-4da4-bf1d-e6274dfd805b\") " pod="kube-system/konnectivity-agent-zqssw" Nov 02 22:51:50.128450 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125647 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-etc-kubernetes\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.128450 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125647 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/668094b0-0e10-47aa-ba49-6944f7083bc6-system-cni-dir\") pod \"multus-additional-cni-plugins-cs2mc\" (UID: \"668094b0-0e10-47aa-ba49-6944f7083bc6\") " pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:51:50.128450 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125671 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-ovn-node-metrics-cert\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.128450 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125723 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-system-cni-dir\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.128450 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125750 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-host-var-lib-kubelet\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.128450 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125796 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-host-var-lib-kubelet\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.128450 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125802 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-system-cni-dir\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.128450 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125838 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/668094b0-0e10-47aa-ba49-6944f7083bc6-cni-binary-copy\") pod \"multus-additional-cni-plugins-cs2mc\" (UID: \"668094b0-0e10-47aa-ba49-6944f7083bc6\") " pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:51:50.128450 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125867 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/668094b0-0e10-47aa-ba49-6944f7083bc6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cs2mc\" (UID: \"668094b0-0e10-47aa-ba49-6944f7083bc6\") " pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:51:50.128450 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125912 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-host-run-k8s-cni-cncf-io\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.128450 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125941 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4shq\" (UniqueName: \"kubernetes.io/projected/668094b0-0e10-47aa-ba49-6944f7083bc6-kube-api-access-k4shq\") pod \"multus-additional-cni-plugins-cs2mc\" (UID: \"668094b0-0e10-47aa-ba49-6944f7083bc6\") " pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:51:50.128450 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125965 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-kubelet\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.128450 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125990 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-ovnkube-script-lib\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.128914 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.125993 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fd96552a-1a78-4fac-8faf-71d36f010905-kubelet-config\") pod \"global-pull-secret-syncer-k4gfs\" (UID: \"fd96552a-1a78-4fac-8faf-71d36f010905\") " pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:51:50.128914 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.126012 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f8b41ab-cbec-4f72-a5d9-853953748c0b-iptables-alerter-script\") pod \"iptables-alerter-92hm7\" (UID: \"4f8b41ab-cbec-4f72-a5d9-853953748c0b\") " pod="openshift-network-operator/iptables-alerter-92hm7" Nov 02 22:51:50.128914 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.126038 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee52c167-6361-4adb-9899-93ca4c2128b8-cni-binary-copy\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.128914 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.126046 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fd96552a-1a78-4fac-8faf-71d36f010905-kubelet-config\") pod \"global-pull-secret-syncer-k4gfs\" (UID: \"fd96552a-1a78-4fac-8faf-71d36f010905\") " pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:51:50.128914 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.126066 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b3753c7-7634-4209-8760-7a52da534549-tmp\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.128914 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.126078 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-host-run-k8s-cni-cncf-io\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.128914 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.126278 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-kubelet\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.128914 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.126381 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/668094b0-0e10-47aa-ba49-6944f7083bc6-cni-binary-copy\") pod \"multus-additional-cni-plugins-cs2mc\" (UID: \"668094b0-0e10-47aa-ba49-6944f7083bc6\") " pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:51:50.128914 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.126115 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/668094b0-0e10-47aa-ba49-6944f7083bc6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cs2mc\" (UID: \"668094b0-0e10-47aa-ba49-6944f7083bc6\") " pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:51:50.128914 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.126604 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-run-systemd\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.128914 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.126620 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkqkc\" (UniqueName: \"kubernetes.io/projected/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-kube-api-access-kkqkc\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.128914 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.126635 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-multus-socket-dir-parent\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.128914 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.126648 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a464bf3-77fa-4fcd-8593-327fa1f4a19a-host\") pod \"node-ca-4mhgx\" (UID: \"3a464bf3-77fa-4fcd-8593-327fa1f4a19a\") " pod="openshift-image-registry/node-ca-4mhgx" Nov 02 22:51:50.128914 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.126691 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a464bf3-77fa-4fcd-8593-327fa1f4a19a-host\") pod \"node-ca-4mhgx\" (UID: \"3a464bf3-77fa-4fcd-8593-327fa1f4a19a\") " pod="openshift-image-registry/node-ca-4mhgx" Nov 02 22:51:50.128914 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.126732 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-run-systemd\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.128914 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.126905 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-multus-socket-dir-parent\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.128914 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.127640 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/668094b0-0e10-47aa-ba49-6944f7083bc6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cs2mc\" (UID: \"668094b0-0e10-47aa-ba49-6944f7083bc6\") " pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:51:50.129343 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.127674 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/668094b0-0e10-47aa-ba49-6944f7083bc6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cs2mc\" (UID: \"668094b0-0e10-47aa-ba49-6944f7083bc6\") " pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:51:50.129343 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.127874 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee52c167-6361-4adb-9899-93ca4c2128b8-cni-binary-copy\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.129343 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.128051 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6b3753c7-7634-4209-8760-7a52da534549-etc-tuned\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.129343 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.128237 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-ovn-node-metrics-cert\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.129343 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.128272 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8fbcc658-b377-4da4-bf1d-e6274dfd805b-agent-certs\") pod \"konnectivity-agent-zqssw\" (UID: \"8fbcc658-b377-4da4-bf1d-e6274dfd805b\") " pod="kube-system/konnectivity-agent-zqssw" Nov 02 22:51:50.129343 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.129321 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b3753c7-7634-4209-8760-7a52da534549-tmp\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.170871 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.170805 2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:51:50.181539 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:50.181511 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:51:50.181539 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:50.181541 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:51:50.181718 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:50.181632 2575 projected.go:196] Error preparing data for projected volume kube-api-access-q7nrp for pod openshift-network-diagnostics/network-check-target-rmgpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:51:50.182294 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:50.182266 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3a3bc57-6813-441c-bdc0-d39eb8214c61-kube-api-access-q7nrp podName:e3a3bc57-6813-441c-bdc0-d39eb8214c61 nodeName:}" failed. No retries permitted until 2026-11-02 22:51:50.68177982 +0000 UTC m=+2.356356619 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-q7nrp" (UniqueName: "kubernetes.io/projected/e3a3bc57-6813-441c-bdc0-d39eb8214c61-kube-api-access-q7nrp") pod "network-check-target-rmgpt" (UID: "e3a3bc57-6813-441c-bdc0-d39eb8214c61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:51:50.183607 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.183584 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctsxp\" (UniqueName: \"kubernetes.io/projected/4f8b41ab-cbec-4f72-a5d9-853953748c0b-kube-api-access-ctsxp\") pod \"iptables-alerter-92hm7\" (UID: \"4f8b41ab-cbec-4f72-a5d9-853953748c0b\") " pod="openshift-network-operator/iptables-alerter-92hm7" Nov 02 22:51:50.184104 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.184079 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvg2k\" (UniqueName: \"kubernetes.io/projected/d3575a78-aad1-41e0-bad4-63724365aeb1-kube-api-access-vvg2k\") pod \"node-resolver-nrxwb\" (UID: \"d3575a78-aad1-41e0-bad4-63724365aeb1\") " pod="openshift-dns/node-resolver-nrxwb" Nov 02 22:51:50.184214 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.184135 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7qr5\" (UniqueName: \"kubernetes.io/projected/ee52c167-6361-4adb-9899-93ca4c2128b8-kube-api-access-q7qr5\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:51:50.184214 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.184144 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6jlv\" (UniqueName: \"kubernetes.io/projected/fc69f07f-4001-4951-aff8-d3d6671cc5d4-kube-api-access-n6jlv\") pod \"network-metrics-daemon-9zqrz\" (UID: \"fc69f07f-4001-4951-aff8-d3d6671cc5d4\") " pod="openshift-multus/network-metrics-daemon-9zqrz" Nov 02 22:51:50.184445 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.184428 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzxgm\" (UniqueName: \"kubernetes.io/projected/6b3753c7-7634-4209-8760-7a52da534549-kube-api-access-wzxgm\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.184527 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.184496 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cml47\" (UniqueName: \"kubernetes.io/projected/3a464bf3-77fa-4fcd-8593-327fa1f4a19a-kube-api-access-cml47\") pod \"node-ca-4mhgx\" (UID: \"3a464bf3-77fa-4fcd-8593-327fa1f4a19a\") " pod="openshift-image-registry/node-ca-4mhgx" Nov 02 22:51:50.185141 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.185118 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4shq\" (UniqueName: \"kubernetes.io/projected/668094b0-0e10-47aa-ba49-6944f7083bc6-kube-api-access-k4shq\") pod \"multus-additional-cni-plugins-cs2mc\" (UID: \"668094b0-0e10-47aa-ba49-6944f7083bc6\") " pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:51:50.185244 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.185153 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkqkc\" (UniqueName: \"kubernetes.io/projected/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-kube-api-access-kkqkc\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.185422 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.185406 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rt8s\" (UniqueName: \"kubernetes.io/projected/cb1743a1-6e7b-4f99-80e6-197a6916580d-kube-api-access-6rt8s\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:51:50.303337 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.303308 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4mhgx" Nov 02 22:51:50.325396 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.325370 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:51:50.327559 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.327540 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zqssw" Nov 02 22:51:50.341303 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.341286 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-92hm7" Nov 02 22:51:50.358040 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.358018 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:51:50.365276 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.365248 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nrxwb" Nov 02 22:51:50.386684 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.386660 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-njn94" Nov 02 22:51:50.399943 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.399917 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:51:50.417672 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.417635 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:51:50.561658 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:50.561607 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2cabf9747a94425892740afb80ec0fc.slice/crio-25637d3d0737f9e52181634fdb879f8dde56f0b2436d7b7e695a500d72cab169 WatchSource:0}: Error finding container 25637d3d0737f9e52181634fdb879f8dde56f0b2436d7b7e695a500d72cab169: Status 404 returned error can't find the container with id 25637d3d0737f9e52181634fdb879f8dde56f0b2436d7b7e695a500d72cab169 Nov 02 22:51:50.562161 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:50.562139 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ba3fed7e8adcc2e1f404604f948f337.slice/crio-edff1bb6209a6e5782a55a8157d2c1884fef8ac84da3480932b04f9c9ebebcec WatchSource:0}: Error finding container edff1bb6209a6e5782a55a8157d2c1884fef8ac84da3480932b04f9c9ebebcec: Status 404 returned error can't find the container with id edff1bb6209a6e5782a55a8157d2c1884fef8ac84da3480932b04f9c9ebebcec Nov 02 22:51:50.567741 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.567725 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:51:50.630965 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.630930 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs\") pod \"network-metrics-daemon-9zqrz\" (UID: \"fc69f07f-4001-4951-aff8-d3d6671cc5d4\") " pod="openshift-multus/network-metrics-daemon-9zqrz" Nov 02 22:51:50.630965 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.630966 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd96552a-1a78-4fac-8faf-71d36f010905-original-pull-secret\") pod \"global-pull-secret-syncer-k4gfs\" (UID: \"fd96552a-1a78-4fac-8faf-71d36f010905\") " pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:51:50.631203 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:50.631057 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:51:50.631203 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:50.631082 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:51:50.631203 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:50.631107 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd96552a-1a78-4fac-8faf-71d36f010905-original-pull-secret podName:fd96552a-1a78-4fac-8faf-71d36f010905 nodeName:}" failed. No retries permitted until 2026-11-02 22:51:51.631095486 +0000 UTC m=+3.305672276 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd96552a-1a78-4fac-8faf-71d36f010905-original-pull-secret") pod "global-pull-secret-syncer-k4gfs" (UID: "fd96552a-1a78-4fac-8faf-71d36f010905") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:51:50.631203 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:50.631142 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs podName:fc69f07f-4001-4951-aff8-d3d6671cc5d4 nodeName:}" failed. No retries permitted until 2026-11-02 22:51:51.631125766 +0000 UTC m=+3.305702557 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs") pod "network-metrics-daemon-9zqrz" (UID: "fc69f07f-4001-4951-aff8-d3d6671cc5d4") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:51:50.732203 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.732172 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7nrp\" (UniqueName: \"kubernetes.io/projected/e3a3bc57-6813-441c-bdc0-d39eb8214c61-kube-api-access-q7nrp\") pod \"network-check-target-rmgpt\" (UID: \"e3a3bc57-6813-441c-bdc0-d39eb8214c61\") " pod="openshift-network-diagnostics/network-check-target-rmgpt" Nov 02 22:51:50.732781 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:50.732387 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:51:50.732781 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:50.732406 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:51:50.732781 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:50.732416 2575 projected.go:196] Error preparing data for projected volume kube-api-access-q7nrp for pod openshift-network-diagnostics/network-check-target-rmgpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:51:50.732781 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:50.732463 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3a3bc57-6813-441c-bdc0-d39eb8214c61-kube-api-access-q7nrp podName:e3a3bc57-6813-441c-bdc0-d39eb8214c61 nodeName:}" failed. No retries permitted until 2026-11-02 22:51:51.732451271 +0000 UTC m=+3.407028066 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-q7nrp" (UniqueName: "kubernetes.io/projected/e3a3bc57-6813-441c-bdc0-d39eb8214c61-kube-api-access-q7nrp") pod "network-check-target-rmgpt" (UID: "e3a3bc57-6813-441c-bdc0-d39eb8214c61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:51:50.764171 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.764145 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:51:50.854769 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.854670 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal" event={"ID":"f2cabf9747a94425892740afb80ec0fc","Type":"ContainerStarted","Data":"25637d3d0737f9e52181634fdb879f8dde56f0b2436d7b7e695a500d72cab169"} Nov 02 22:51:50.855637 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.855617 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-211.us-west-2.compute.internal" event={"ID":"7ba3fed7e8adcc2e1f404604f948f337","Type":"ContainerStarted","Data":"edff1bb6209a6e5782a55a8157d2c1884fef8ac84da3480932b04f9c9ebebcec"} Nov 02 22:51:50.968084 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:50.968050 2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:51:51.114690 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:51.114582 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 22:46:49 +0000 UTC" deadline="2028-04-05 22:50:17.289001991 +0000 UTC" Nov 02 22:51:51.114690 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:51.114625 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12479h58m26.174379188s" Nov 02 22:51:51.180414 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:51.180386 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f8b41ab_cbec_4f72_a5d9_853953748c0b.slice/crio-9ef782737103253b57fdf6f5c12cb642b00ca83215d5a52e554ce2f17e691f49 WatchSource:0}: Error finding container 9ef782737103253b57fdf6f5c12cb642b00ca83215d5a52e554ce2f17e691f49: Status 404 returned error can't find the container with id 9ef782737103253b57fdf6f5c12cb642b00ca83215d5a52e554ce2f17e691f49 Nov 02 22:51:51.238173 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:51.238146 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee52c167_6361_4adb_9899_93ca4c2128b8.slice/crio-25ce3b94c1b208749a7a75bead39555de3944bf63e6d8f360ca695541ce8f157 WatchSource:0}: Error finding container 25ce3b94c1b208749a7a75bead39555de3944bf63e6d8f360ca695541ce8f157: Status 404 returned error can't find the container with id 25ce3b94c1b208749a7a75bead39555de3944bf63e6d8f360ca695541ce8f157 Nov 02 22:51:51.248235 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:51.248212 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb1743a1_6e7b_4f99_80e6_197a6916580d.slice/crio-58436ee67e99f72594ebf75d78e03a97fbc28924da7c08d072582970e23d689e WatchSource:0}: Error finding container 58436ee67e99f72594ebf75d78e03a97fbc28924da7c08d072582970e23d689e: Status 404 returned error can't find the container with id 58436ee67e99f72594ebf75d78e03a97fbc28924da7c08d072582970e23d689e Nov 02 22:51:51.255194 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:51.255162 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3575a78_aad1_41e0_bad4_63724365aeb1.slice/crio-5d546ba5bc54172ecbea70ee0fbfebe48033ebf77bc7a94c95663240295e4292 WatchSource:0}: Error finding container 5d546ba5bc54172ecbea70ee0fbfebe48033ebf77bc7a94c95663240295e4292: Status 404 returned error can't find the container with id 5d546ba5bc54172ecbea70ee0fbfebe48033ebf77bc7a94c95663240295e4292 Nov 02 22:51:51.262696 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:51.262674 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fbcc658_b377_4da4_bf1d_e6274dfd805b.slice/crio-914d69658f5c63ff96087f02576c562a6e1167cc1722fef6a93a4c62118ab3ab WatchSource:0}: Error finding container 914d69658f5c63ff96087f02576c562a6e1167cc1722fef6a93a4c62118ab3ab: Status 404 returned error can't find the container with id 914d69658f5c63ff96087f02576c562a6e1167cc1722fef6a93a4c62118ab3ab Nov 02 22:51:51.298500 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:51.298472 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod324d3fbb_66cd_4e3f_8cc1_f65ce06c0d49.slice/crio-4f48f38b5e24171bd945b75111ee6fa5347a41c456770b64e8f7d43d1ee65e38 WatchSource:0}: Error finding container 4f48f38b5e24171bd945b75111ee6fa5347a41c456770b64e8f7d43d1ee65e38: Status 404 returned error can't find the container with id 4f48f38b5e24171bd945b75111ee6fa5347a41c456770b64e8f7d43d1ee65e38 Nov 02 22:51:51.299113 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:51.299090 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b3753c7_7634_4209_8760_7a52da534549.slice/crio-cd66118a9af2036e394f7ef9aa55afebc41410ff4920958d5388c97db4388644 WatchSource:0}: Error finding container cd66118a9af2036e394f7ef9aa55afebc41410ff4920958d5388c97db4388644: Status 404 returned error can't find the container with id cd66118a9af2036e394f7ef9aa55afebc41410ff4920958d5388c97db4388644 Nov 02 22:51:51.302938 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:51.302917 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a464bf3_77fa_4fcd_8593_327fa1f4a19a.slice/crio-a7eef4c62c90023e339d07ef34aa6a936384058d68431afa17a67efc0579b03e WatchSource:0}: Error finding container a7eef4c62c90023e339d07ef34aa6a936384058d68431afa17a67efc0579b03e: Status 404 returned error can't find the container with id a7eef4c62c90023e339d07ef34aa6a936384058d68431afa17a67efc0579b03e Nov 02 22:51:51.346310 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:51:51.346283 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod668094b0_0e10_47aa_ba49_6944f7083bc6.slice/crio-eb0cdddb2bd55368782469edc79d2349ea253148a8e8af8a37591b6f4181ba61 WatchSource:0}: Error finding container eb0cdddb2bd55368782469edc79d2349ea253148a8e8af8a37591b6f4181ba61: Status 404 returned error can't find the container with id eb0cdddb2bd55368782469edc79d2349ea253148a8e8af8a37591b6f4181ba61 Nov 02 22:51:51.639664 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:51.639627 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs\") pod \"network-metrics-daemon-9zqrz\" (UID: \"fc69f07f-4001-4951-aff8-d3d6671cc5d4\") " pod="openshift-multus/network-metrics-daemon-9zqrz" Nov 02 22:51:51.639664 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:51.639668 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd96552a-1a78-4fac-8faf-71d36f010905-original-pull-secret\") pod \"global-pull-secret-syncer-k4gfs\" (UID: \"fd96552a-1a78-4fac-8faf-71d36f010905\") " pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:51:51.639942 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:51.639763 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:51:51.639942 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:51.639787 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:51:51.639942 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:51.639818 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd96552a-1a78-4fac-8faf-71d36f010905-original-pull-secret podName:fd96552a-1a78-4fac-8faf-71d36f010905 nodeName:}" failed. No retries permitted until 2026-11-02 22:51:53.639801327 +0000 UTC m=+5.314378106 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd96552a-1a78-4fac-8faf-71d36f010905-original-pull-secret") pod "global-pull-secret-syncer-k4gfs" (UID: "fd96552a-1a78-4fac-8faf-71d36f010905") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:51:51.639942 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:51.639845 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs podName:fc69f07f-4001-4951-aff8-d3d6671cc5d4 nodeName:}" failed. No retries permitted until 2026-11-02 22:51:53.639828939 +0000 UTC m=+5.314405732 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs") pod "network-metrics-daemon-9zqrz" (UID: "fc69f07f-4001-4951-aff8-d3d6671cc5d4") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:51:51.740660 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:51.740628 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7nrp\" (UniqueName: \"kubernetes.io/projected/e3a3bc57-6813-441c-bdc0-d39eb8214c61-kube-api-access-q7nrp\") pod \"network-check-target-rmgpt\" (UID: \"e3a3bc57-6813-441c-bdc0-d39eb8214c61\") " pod="openshift-network-diagnostics/network-check-target-rmgpt" Nov 02 22:51:51.741120 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:51.740777 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:51:51.741120 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:51.740795 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:51:51.741120 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:51.740805 2575 projected.go:196] Error preparing data for projected volume kube-api-access-q7nrp for pod openshift-network-diagnostics/network-check-target-rmgpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:51:51.741120 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:51.740851 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3a3bc57-6813-441c-bdc0-d39eb8214c61-kube-api-access-q7nrp podName:e3a3bc57-6813-441c-bdc0-d39eb8214c61 nodeName:}" failed. No retries permitted until 2026-11-02 22:51:53.740838731 +0000 UTC m=+5.415415510 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-q7nrp" (UniqueName: "kubernetes.io/projected/e3a3bc57-6813-441c-bdc0-d39eb8214c61-kube-api-access-q7nrp") pod "network-check-target-rmgpt" (UID: "e3a3bc57-6813-441c-bdc0-d39eb8214c61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:51:51.851148 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:51.851117 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:51:51.851314 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:51.851126 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zqrz" Nov 02 22:51:51.851314 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:51.851235 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rmgpt" Nov 02 22:51:51.851314 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:51.851236 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4gfs" podUID="fd96552a-1a78-4fac-8faf-71d36f010905" Nov 02 22:51:51.851487 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:51.851340 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zqrz" podUID="fc69f07f-4001-4951-aff8-d3d6671cc5d4" Nov 02 22:51:51.851487 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:51.851438 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rmgpt" podUID="e3a3bc57-6813-441c-bdc0-d39eb8214c61" Nov 02 22:51:51.858687 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:51.858650 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" event={"ID":"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49","Type":"ContainerStarted","Data":"4f48f38b5e24171bd945b75111ee6fa5347a41c456770b64e8f7d43d1ee65e38"} Nov 02 22:51:51.860030 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:51.860004 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zqssw" event={"ID":"8fbcc658-b377-4da4-bf1d-e6274dfd805b","Type":"ContainerStarted","Data":"914d69658f5c63ff96087f02576c562a6e1167cc1722fef6a93a4c62118ab3ab"} Nov 02 22:51:51.861213 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:51.861187 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" event={"ID":"cb1743a1-6e7b-4f99-80e6-197a6916580d","Type":"ContainerStarted","Data":"58436ee67e99f72594ebf75d78e03a97fbc28924da7c08d072582970e23d689e"} Nov 02 22:51:51.862411 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:51.862368 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4mhgx" event={"ID":"3a464bf3-77fa-4fcd-8593-327fa1f4a19a","Type":"ContainerStarted","Data":"a7eef4c62c90023e339d07ef34aa6a936384058d68431afa17a67efc0579b03e"} Nov 02 22:51:51.863549 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:51.863524 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nrxwb" event={"ID":"d3575a78-aad1-41e0-bad4-63724365aeb1","Type":"ContainerStarted","Data":"5d546ba5bc54172ecbea70ee0fbfebe48033ebf77bc7a94c95663240295e4292"} Nov 02 22:51:51.864647 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:51.864623 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-njn94" event={"ID":"ee52c167-6361-4adb-9899-93ca4c2128b8","Type":"ContainerStarted","Data":"25ce3b94c1b208749a7a75bead39555de3944bf63e6d8f360ca695541ce8f157"} Nov 02 22:51:51.865819 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:51.865795 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-92hm7" event={"ID":"4f8b41ab-cbec-4f72-a5d9-853953748c0b","Type":"ContainerStarted","Data":"9ef782737103253b57fdf6f5c12cb642b00ca83215d5a52e554ce2f17e691f49"} Nov 02 22:51:51.867101 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:51.867069 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cs2mc" event={"ID":"668094b0-0e10-47aa-ba49-6944f7083bc6","Type":"ContainerStarted","Data":"eb0cdddb2bd55368782469edc79d2349ea253148a8e8af8a37591b6f4181ba61"} Nov 02 22:51:51.868198 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:51.868179 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" event={"ID":"6b3753c7-7634-4209-8760-7a52da534549","Type":"ContainerStarted","Data":"cd66118a9af2036e394f7ef9aa55afebc41410ff4920958d5388c97db4388644"} Nov 02 22:51:52.115127 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:52.115021 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 22:46:49 +0000 UTC" deadline="2028-04-12 10:47:25.237414348 +0000 UTC" Nov 02 22:51:52.115127 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:52.115074 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12635h55m33.122344342s" Nov 02 22:51:53.660573 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:53.660535 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs\") pod \"network-metrics-daemon-9zqrz\" (UID: \"fc69f07f-4001-4951-aff8-d3d6671cc5d4\") " pod="openshift-multus/network-metrics-daemon-9zqrz" Nov 02 22:51:53.661113 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:53.660592 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd96552a-1a78-4fac-8faf-71d36f010905-original-pull-secret\") pod \"global-pull-secret-syncer-k4gfs\" (UID: \"fd96552a-1a78-4fac-8faf-71d36f010905\") " pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:51:53.661113 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:53.660763 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:51:53.661113 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:53.660826 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd96552a-1a78-4fac-8faf-71d36f010905-original-pull-secret podName:fd96552a-1a78-4fac-8faf-71d36f010905 nodeName:}" failed. No retries permitted until 2026-11-02 22:51:57.660807277 +0000 UTC m=+9.335384069 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd96552a-1a78-4fac-8faf-71d36f010905-original-pull-secret") pod "global-pull-secret-syncer-k4gfs" (UID: "fd96552a-1a78-4fac-8faf-71d36f010905") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:51:53.661113 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:53.660997 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:51:53.661113 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:53.661052 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs podName:fc69f07f-4001-4951-aff8-d3d6671cc5d4 nodeName:}" failed. No retries permitted until 2026-11-02 22:51:57.661038009 +0000 UTC m=+9.335614803 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs") pod "network-metrics-daemon-9zqrz" (UID: "fc69f07f-4001-4951-aff8-d3d6671cc5d4") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:51:53.761567 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:53.761527 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7nrp\" (UniqueName: \"kubernetes.io/projected/e3a3bc57-6813-441c-bdc0-d39eb8214c61-kube-api-access-q7nrp\") pod \"network-check-target-rmgpt\" (UID: \"e3a3bc57-6813-441c-bdc0-d39eb8214c61\") " pod="openshift-network-diagnostics/network-check-target-rmgpt" Nov 02 22:51:53.761752 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:53.761694 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:51:53.761752 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:53.761716 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:51:53.761752 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:53.761727 2575 projected.go:196] Error preparing data for projected volume kube-api-access-q7nrp for pod openshift-network-diagnostics/network-check-target-rmgpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:51:53.761931 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:53.761784 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3a3bc57-6813-441c-bdc0-d39eb8214c61-kube-api-access-q7nrp podName:e3a3bc57-6813-441c-bdc0-d39eb8214c61 nodeName:}" failed. No retries permitted until 2026-11-02 22:51:57.761767073 +0000 UTC m=+9.436343877 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-q7nrp" (UniqueName: "kubernetes.io/projected/e3a3bc57-6813-441c-bdc0-d39eb8214c61-kube-api-access-q7nrp") pod "network-check-target-rmgpt" (UID: "e3a3bc57-6813-441c-bdc0-d39eb8214c61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:51:53.851376 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:53.851245 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:51:53.851573 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:53.851443 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4gfs" podUID="fd96552a-1a78-4fac-8faf-71d36f010905" Nov 02 22:51:53.852123 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:53.851870 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rmgpt" Nov 02 22:51:53.852123 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:53.851987 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rmgpt" podUID="e3a3bc57-6813-441c-bdc0-d39eb8214c61" Nov 02 22:51:53.852123 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:53.852039 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zqrz" Nov 02 22:51:53.852123 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:53.852117 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zqrz" podUID="fc69f07f-4001-4951-aff8-d3d6671cc5d4" Nov 02 22:51:55.851871 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:55.851460 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zqrz" Nov 02 22:51:55.851871 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:55.851711 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:51:55.851871 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:55.851829 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4gfs" podUID="fd96552a-1a78-4fac-8faf-71d36f010905" Nov 02 22:51:55.852558 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:55.851923 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zqrz" podUID="fc69f07f-4001-4951-aff8-d3d6671cc5d4" Nov 02 22:51:55.852558 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:55.851996 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rmgpt" Nov 02 22:51:55.852558 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:55.852069 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rmgpt" podUID="e3a3bc57-6813-441c-bdc0-d39eb8214c61" Nov 02 22:51:55.888432 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:55.888402 2575 generic.go:356] "Generic (PLEG): container finished" podID="668094b0-0e10-47aa-ba49-6944f7083bc6" containerID="fecfdf9c325b12d132bba90fb5eba4090ea17eec3d79bd6e5d3e9406fcad0d9c" exitCode=0 Nov 02 22:51:55.888661 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:55.888507 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cs2mc" event={"ID":"668094b0-0e10-47aa-ba49-6944f7083bc6","Type":"ContainerDied","Data":"fecfdf9c325b12d132bba90fb5eba4090ea17eec3d79bd6e5d3e9406fcad0d9c"} Nov 02 22:51:55.896705 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:55.896672 2575 generic.go:356] "Generic (PLEG): container finished" podID="f2cabf9747a94425892740afb80ec0fc" containerID="ca14beba5a668986c160950e31521f74eeb8b3baf0c8ac82db04fa420f1dc44b" exitCode=0 Nov 02 22:51:55.896800 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:55.896752 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal" event={"ID":"f2cabf9747a94425892740afb80ec0fc","Type":"ContainerDied","Data":"ca14beba5a668986c160950e31521f74eeb8b3baf0c8ac82db04fa420f1dc44b"} Nov 02 22:51:55.904875 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:55.904412 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-211.us-west-2.compute.internal" event={"ID":"7ba3fed7e8adcc2e1f404604f948f337","Type":"ContainerStarted","Data":"3c7e8a2b873bfc147557449a3285c21c8bb11eb9626f224d770ad62a61319e05"} Nov 02 22:51:56.176364 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:56.176293 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-211.us-west-2.compute.internal" podStartSLOduration=7.176275334 podStartE2EDuration="7.176275334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:51:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:51:56.175171271 +0000 UTC m=+7.849748072" watchObservedRunningTime="2026-11-02 22:51:56.176275334 +0000 UTC m=+7.850852135" Nov 02 22:51:56.913369 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:56.912658 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zqssw" event={"ID":"8fbcc658-b377-4da4-bf1d-e6274dfd805b","Type":"ContainerStarted","Data":"297e36463528f94a07a309033d45c3c1bc20fd07e3379379e7b3c6119859f52e"} Nov 02 22:51:56.917493 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:56.917419 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal" event={"ID":"f2cabf9747a94425892740afb80ec0fc","Type":"ContainerStarted","Data":"5e34994a75d18bb5dec964ac44a1468f4db7795cbcd1467ada7e6e4fd1d58783"} Nov 02 22:51:57.147521 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:57.147441 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zqssw" podStartSLOduration=3.5453121 podStartE2EDuration="8.147420365s" totalImagesPullingTime="4.602108265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:51:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:51:57.030522089 +0000 UTC m=+8.705098891" watchObservedRunningTime="2026-11-02 22:51:57.147420365 +0000 UTC m=+8.821997166" Nov 02 22:51:57.483435 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:57.483352 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zqssw" Nov 02 22:51:57.483820 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:57.483802 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zqssw" Nov 02 22:51:57.600292 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:57.600231 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal" podStartSLOduration=8.600208761 podStartE2EDuration="8.600208761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:51:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:51:57.148428672 +0000 UTC m=+8.823005473" watchObservedRunningTime="2026-11-02 22:51:57.600208761 +0000 UTC m=+9.274785578" Nov 02 22:51:57.698506 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:57.697854 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs\") pod \"network-metrics-daemon-9zqrz\" (UID: \"fc69f07f-4001-4951-aff8-d3d6671cc5d4\") " pod="openshift-multus/network-metrics-daemon-9zqrz" Nov 02 22:51:57.698506 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:57.697971 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd96552a-1a78-4fac-8faf-71d36f010905-original-pull-secret\") pod \"global-pull-secret-syncer-k4gfs\" (UID: \"fd96552a-1a78-4fac-8faf-71d36f010905\") " pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:51:57.698506 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:57.698107 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:51:57.698506 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:57.698162 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd96552a-1a78-4fac-8faf-71d36f010905-original-pull-secret podName:fd96552a-1a78-4fac-8faf-71d36f010905 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:05.69814606 +0000 UTC m=+17.372722858 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd96552a-1a78-4fac-8faf-71d36f010905-original-pull-secret") pod "global-pull-secret-syncer-k4gfs" (UID: "fd96552a-1a78-4fac-8faf-71d36f010905") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:51:57.698843 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:57.698629 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:51:57.698843 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:57.698727 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs podName:fc69f07f-4001-4951-aff8-d3d6671cc5d4 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:05.698709944 +0000 UTC m=+17.373286725 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs") pod "network-metrics-daemon-9zqrz" (UID: "fc69f07f-4001-4951-aff8-d3d6671cc5d4") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:51:57.799279 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:57.799235 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7nrp\" (UniqueName: \"kubernetes.io/projected/e3a3bc57-6813-441c-bdc0-d39eb8214c61-kube-api-access-q7nrp\") pod \"network-check-target-rmgpt\" (UID: \"e3a3bc57-6813-441c-bdc0-d39eb8214c61\") " pod="openshift-network-diagnostics/network-check-target-rmgpt" Nov 02 22:51:57.799467 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:57.799448 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:51:57.799571 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:57.799482 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:51:57.799571 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:57.799493 2575 projected.go:196] Error preparing data for projected volume kube-api-access-q7nrp for pod openshift-network-diagnostics/network-check-target-rmgpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:51:57.799571 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:57.799538 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3a3bc57-6813-441c-bdc0-d39eb8214c61-kube-api-access-q7nrp podName:e3a3bc57-6813-441c-bdc0-d39eb8214c61 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:05.799524006 +0000 UTC m=+17.474100797 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-q7nrp" (UniqueName: "kubernetes.io/projected/e3a3bc57-6813-441c-bdc0-d39eb8214c61-kube-api-access-q7nrp") pod "network-check-target-rmgpt" (UID: "e3a3bc57-6813-441c-bdc0-d39eb8214c61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:51:57.851425 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:57.851394 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:51:57.851583 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:57.851507 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4gfs" podUID="fd96552a-1a78-4fac-8faf-71d36f010905" Nov 02 22:51:57.851583 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:57.851570 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rmgpt" Nov 02 22:51:57.851708 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:57.851621 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rmgpt" podUID="e3a3bc57-6813-441c-bdc0-d39eb8214c61" Nov 02 22:51:57.851708 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:57.851660 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zqrz" Nov 02 22:51:57.851708 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:57.851707 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zqrz" podUID="fc69f07f-4001-4951-aff8-d3d6671cc5d4" Nov 02 22:51:57.924644 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:57.923729 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4mhgx" event={"ID":"3a464bf3-77fa-4fcd-8593-327fa1f4a19a","Type":"ContainerStarted","Data":"43f48e794b32d8444df7234f7db92cefb5415a6eab28c82c103b51f2e3b62f80"} Nov 02 22:51:58.927513 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:58.927467 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" event={"ID":"cb1743a1-6e7b-4f99-80e6-197a6916580d","Type":"ContainerStarted","Data":"f60562abcb5d5775f93e57c46d8e31681dab91c2094b26638d0da5a791a37f8b"} Nov 02 22:51:58.927971 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:58.927716 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 22:51:59.851669 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:59.851626 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zqrz" Nov 02 22:51:59.851846 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:59.851749 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zqrz" podUID="fc69f07f-4001-4951-aff8-d3d6671cc5d4" Nov 02 22:51:59.852138 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:59.852121 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rmgpt" Nov 02 22:51:59.852228 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:59.852209 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rmgpt" podUID="e3a3bc57-6813-441c-bdc0-d39eb8214c61" Nov 02 22:51:59.852281 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:51:59.852265 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:51:59.852368 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:51:59.852341 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4gfs" podUID="fd96552a-1a78-4fac-8faf-71d36f010905" Nov 02 22:52:00.253090 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:00.252963 2575 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 22:52:00.803362 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:00.803009 2575 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T22:52:00.252985013Z","UUID":"bbb36bb4-2b3d-4d51-8a3d-3f4e7ef3dc9f","Handler":null,"Name":"","Endpoint":""} Nov 02 22:52:00.804676 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:00.804649 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 22:52:00.804785 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:00.804681 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 22:52:00.934830 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:00.934794 2575 generic.go:356] "Generic (PLEG): container finished" podID="668094b0-0e10-47aa-ba49-6944f7083bc6" containerID="8961f820b351f2772f7c0b9d7090d9e8aabd6904e744079229a03bef2f5fd88d" exitCode=0 Nov 02 22:52:00.935018 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:00.934873 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cs2mc" event={"ID":"668094b0-0e10-47aa-ba49-6944f7083bc6","Type":"ContainerDied","Data":"8961f820b351f2772f7c0b9d7090d9e8aabd6904e744079229a03bef2f5fd88d"} Nov 02 22:52:00.939186 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:00.939116 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" event={"ID":"cb1743a1-6e7b-4f99-80e6-197a6916580d","Type":"ContainerStarted","Data":"f630b215028ac4ace67cc00cde2de386324259f3a29b2658efd7ad406d77bd83"} Nov 02 22:52:00.941725 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:00.941698 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nrxwb" event={"ID":"d3575a78-aad1-41e0-bad4-63724365aeb1","Type":"ContainerStarted","Data":"de14f698307b65accffd64d1694d7ed59191ea546803b1041e96a3cffb318e93"} Nov 02 22:52:00.943217 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:00.943194 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-njn94" event={"ID":"ee52c167-6361-4adb-9899-93ca4c2128b8","Type":"ContainerStarted","Data":"29da5dd1f4641b16d057b3e7223addcd5daaa11f9210350eecfb48a760c03e51"} Nov 02 22:52:01.055869 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:01.055818 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4mhgx" podStartSLOduration=7.349047584 podStartE2EDuration="13.055805673s" totalImagesPullingTime="5.706758089s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:51:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:51:58.039856088 +0000 UTC m=+9.714432893" watchObservedRunningTime="2026-11-02 22:52:01.055805673 +0000 UTC m=+12.730382520" Nov 02 22:52:01.170508 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:01.170418 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-nrxwb" podStartSLOduration=3.555075107 podStartE2EDuration="12.170408481s" totalImagesPullingTime="8.615333374s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:51:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:01.170131286 +0000 UTC m=+12.844708087" watchObservedRunningTime="2026-11-02 22:52:01.170408481 +0000 UTC m=+12.844985282" Nov 02 22:52:01.289850 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:01.289789 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-njn94" podStartSLOduration=3.640016702 podStartE2EDuration="12.289770152s" totalImagesPullingTime="8.64975345s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:51:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:01.288732692 +0000 UTC m=+12.963309491" watchObservedRunningTime="2026-11-02 22:52:01.289770152 +0000 UTC m=+12.964346953" Nov 02 22:52:01.851655 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:01.851629 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zqrz" Nov 02 22:52:01.851804 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:01.851765 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:52:01.851804 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:01.851760 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zqrz" podUID="fc69f07f-4001-4951-aff8-d3d6671cc5d4" Nov 02 22:52:01.851951 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:01.851863 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4gfs" podUID="fd96552a-1a78-4fac-8faf-71d36f010905" Nov 02 22:52:01.851951 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:01.851877 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rmgpt" Nov 02 22:52:01.852036 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:01.851995 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rmgpt" podUID="e3a3bc57-6813-441c-bdc0-d39eb8214c61" Nov 02 22:52:01.948119 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:01.948069 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" event={"ID":"cb1743a1-6e7b-4f99-80e6-197a6916580d","Type":"ContainerStarted","Data":"6ee2383d6553c26578f57595374cedf37303f4256415e0831837452d403a7990"} Nov 02 22:52:01.949968 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:01.949936 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-92hm7" event={"ID":"4f8b41ab-cbec-4f72-a5d9-853953748c0b","Type":"ContainerStarted","Data":"56f62a81b7e0f1ad132501fc07896421e253be5925815f407e5dd5ad1a97dd7b"} Nov 02 22:52:02.186986 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:02.186880 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" podStartSLOduration=4.004462223 podStartE2EDuration="14.186864634s" totalImagesPullingTime="10.182402411s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:51:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:02.070543657 +0000 UTC m=+13.745120436" watchObservedRunningTime="2026-11-02 22:52:02.186864634 +0000 UTC m=+13.861441434" Nov 02 22:52:02.187219 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:02.187199 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-92hm7" podStartSLOduration=4.310776525 podStartE2EDuration="13.187191903s" totalImagesPullingTime="8.876415378s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:51:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:02.186541438 +0000 UTC m=+13.861118240" watchObservedRunningTime="2026-11-02 22:52:02.187191903 +0000 UTC m=+13.861768703" Nov 02 22:52:02.954519 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:02.954437 2575 generic.go:356] "Generic (PLEG): container finished" podID="668094b0-0e10-47aa-ba49-6944f7083bc6" containerID="656a1019ab4fada7b0d0b3842f59e447ba28bd247e528b012a90be4c49860907" exitCode=0 Nov 02 22:52:02.955317 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:02.954540 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cs2mc" event={"ID":"668094b0-0e10-47aa-ba49-6944f7083bc6","Type":"ContainerDied","Data":"656a1019ab4fada7b0d0b3842f59e447ba28bd247e528b012a90be4c49860907"} Nov 02 22:52:02.956758 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:02.956734 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" event={"ID":"6b3753c7-7634-4209-8760-7a52da534549","Type":"ContainerStarted","Data":"34ebcccd6c5131931d4999923adc65aaaa85dc979e78065b1439ad6d6160a412"} Nov 02 22:52:03.297677 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:03.297577 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" podStartSLOduration=3.637058996 podStartE2EDuration="14.297559322s" totalImagesPullingTime="10.660500326s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:51:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:03.297424218 +0000 UTC m=+14.972001019" watchObservedRunningTime="2026-11-02 22:52:03.297559322 +0000 UTC m=+14.972136124" Nov 02 22:52:03.850854 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:03.850817 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rmgpt" Nov 02 22:52:03.851057 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:03.850864 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zqrz" Nov 02 22:52:03.851057 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:03.850817 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:52:03.851057 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:03.850962 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rmgpt" podUID="e3a3bc57-6813-441c-bdc0-d39eb8214c61" Nov 02 22:52:03.851057 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:03.851028 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zqrz" podUID="fc69f07f-4001-4951-aff8-d3d6671cc5d4" Nov 02 22:52:03.851270 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:03.851079 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4gfs" podUID="fd96552a-1a78-4fac-8faf-71d36f010905" Nov 02 22:52:04.967127 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:04.966857 2575 generic.go:356] "Generic (PLEG): container finished" podID="668094b0-0e10-47aa-ba49-6944f7083bc6" containerID="56c6e25b7361e6d92201ce0315b72296ea92eeb44bfd1c90b437f68292f4c716" exitCode=0 Nov 02 22:52:04.967127 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:04.966925 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cs2mc" event={"ID":"668094b0-0e10-47aa-ba49-6944f7083bc6","Type":"ContainerDied","Data":"56c6e25b7361e6d92201ce0315b72296ea92eeb44bfd1c90b437f68292f4c716"} Nov 02 22:52:05.763835 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:05.763785 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs\") pod \"network-metrics-daemon-9zqrz\" (UID: \"fc69f07f-4001-4951-aff8-d3d6671cc5d4\") " pod="openshift-multus/network-metrics-daemon-9zqrz" Nov 02 22:52:05.763835 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:05.763829 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd96552a-1a78-4fac-8faf-71d36f010905-original-pull-secret\") pod \"global-pull-secret-syncer-k4gfs\" (UID: \"fd96552a-1a78-4fac-8faf-71d36f010905\") " pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:52:05.764088 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:05.763947 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:52:05.764088 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:05.763955 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:52:05.764088 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:05.764009 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd96552a-1a78-4fac-8faf-71d36f010905-original-pull-secret podName:fd96552a-1a78-4fac-8faf-71d36f010905 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:21.763996858 +0000 UTC m=+33.438573636 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd96552a-1a78-4fac-8faf-71d36f010905-original-pull-secret") pod "global-pull-secret-syncer-k4gfs" (UID: "fd96552a-1a78-4fac-8faf-71d36f010905") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:52:05.764088 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:05.764025 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs podName:fc69f07f-4001-4951-aff8-d3d6671cc5d4 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:21.764018513 +0000 UTC m=+33.438595291 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs") pod "network-metrics-daemon-9zqrz" (UID: "fc69f07f-4001-4951-aff8-d3d6671cc5d4") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:52:05.851466 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:05.851437 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:52:05.851650 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:05.851480 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zqrz" Nov 02 22:52:05.851650 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:05.851569 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4gfs" podUID="fd96552a-1a78-4fac-8faf-71d36f010905" Nov 02 22:52:05.851650 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:05.851648 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zqrz" podUID="fc69f07f-4001-4951-aff8-d3d6671cc5d4" Nov 02 22:52:05.851757 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:05.851683 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rmgpt" Nov 02 22:52:05.851757 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:05.851723 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rmgpt" podUID="e3a3bc57-6813-441c-bdc0-d39eb8214c61" Nov 02 22:52:05.864662 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:05.864626 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7nrp\" (UniqueName: \"kubernetes.io/projected/e3a3bc57-6813-441c-bdc0-d39eb8214c61-kube-api-access-q7nrp\") pod \"network-check-target-rmgpt\" (UID: \"e3a3bc57-6813-441c-bdc0-d39eb8214c61\") " pod="openshift-network-diagnostics/network-check-target-rmgpt" Nov 02 22:52:05.864814 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:05.864795 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:52:05.864862 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:05.864822 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:52:05.864862 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:05.864833 2575 projected.go:196] Error preparing data for projected volume kube-api-access-q7nrp for pod openshift-network-diagnostics/network-check-target-rmgpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:52:05.864971 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:05.864906 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3a3bc57-6813-441c-bdc0-d39eb8214c61-kube-api-access-q7nrp podName:e3a3bc57-6813-441c-bdc0-d39eb8214c61 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:21.864874248 +0000 UTC m=+33.539451051 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-q7nrp" (UniqueName: "kubernetes.io/projected/e3a3bc57-6813-441c-bdc0-d39eb8214c61-kube-api-access-q7nrp") pod "network-check-target-rmgpt" (UID: "e3a3bc57-6813-441c-bdc0-d39eb8214c61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:52:06.975455 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:06.975210 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" event={"ID":"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49","Type":"ContainerStarted","Data":"bc315dc016b1100d003fb3612aa3630480d388866c11dcbd8f9f4a8a2af835c9"} Nov 02 22:52:06.975852 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:06.975456 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" event={"ID":"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49","Type":"ContainerStarted","Data":"c44e9206581aacccd52966226ca37e951172de71288c15008ced72615dd33791"} Nov 02 22:52:06.975852 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:06.975505 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" event={"ID":"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49","Type":"ContainerStarted","Data":"f1e852559387c0e5c01c8d1e30c21713f0e3936159c044164d77d10548ebe8b4"} Nov 02 22:52:06.975852 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:06.975558 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" event={"ID":"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49","Type":"ContainerStarted","Data":"e20ddb90a1104174c2fbbb97101227752cc2803cb5ad08ac01a0eb1282286700"} Nov 02 22:52:06.975852 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:06.975596 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" event={"ID":"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49","Type":"ContainerStarted","Data":"c4bcc7f9942b1125c1b23fc9dce49ca72c2ac591f62d02e876372816cd17dea4"} Nov 02 22:52:06.975852 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:06.975622 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" event={"ID":"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49","Type":"ContainerStarted","Data":"782b0315ef33ac06e989829960a27203cda5ee262cb685a1a28cb51803472ee2"} Nov 02 22:52:07.851743 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:07.851661 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rmgpt" Nov 02 22:52:07.851743 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:07.851689 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:52:07.851988 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:07.851760 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rmgpt" podUID="e3a3bc57-6813-441c-bdc0-d39eb8214c61" Nov 02 22:52:07.851988 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:07.851810 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zqrz" Nov 02 22:52:07.851988 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:07.851946 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zqrz" podUID="fc69f07f-4001-4951-aff8-d3d6671cc5d4" Nov 02 22:52:07.852099 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:07.851985 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4gfs" podUID="fd96552a-1a78-4fac-8faf-71d36f010905" Nov 02 22:52:07.979680 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:07.979652 2575 generic.go:356] "Generic (PLEG): container finished" podID="668094b0-0e10-47aa-ba49-6944f7083bc6" containerID="49b3f268de56c791d565646fb91148756abe602a63dfe4e49d4be1bd2d457cec" exitCode=0 Nov 02 22:52:07.979680 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:07.979685 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cs2mc" event={"ID":"668094b0-0e10-47aa-ba49-6944f7083bc6","Type":"ContainerDied","Data":"49b3f268de56c791d565646fb91148756abe602a63dfe4e49d4be1bd2d457cec"} Nov 02 22:52:08.984778 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:08.984694 2575 generic.go:356] "Generic (PLEG): container finished" podID="668094b0-0e10-47aa-ba49-6944f7083bc6" containerID="de97d17d30df0dc6968c21c9462490f96b52ce1d0365908428ab462c3f30063b" exitCode=0 Nov 02 22:52:08.985315 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:08.984784 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cs2mc" event={"ID":"668094b0-0e10-47aa-ba49-6944f7083bc6","Type":"ContainerDied","Data":"de97d17d30df0dc6968c21c9462490f96b52ce1d0365908428ab462c3f30063b"} Nov 02 22:52:08.988273 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:08.988203 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" event={"ID":"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49","Type":"ContainerStarted","Data":"5abf051470836f28d9196b3c6e7656e078fbb87f407ede66a15c7177fe058fb2"} Nov 02 22:52:09.850997 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:09.850967 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:52:09.851140 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:09.851033 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zqrz" Nov 02 22:52:09.851140 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:09.851124 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4gfs" podUID="fd96552a-1a78-4fac-8faf-71d36f010905" Nov 02 22:52:09.851140 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:09.851131 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rmgpt" Nov 02 22:52:09.851258 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:09.851201 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zqrz" podUID="fc69f07f-4001-4951-aff8-d3d6671cc5d4" Nov 02 22:52:09.851301 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:09.851273 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rmgpt" podUID="e3a3bc57-6813-441c-bdc0-d39eb8214c61" Nov 02 22:52:09.949568 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:09.949532 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zqssw" Nov 02 22:52:09.949725 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:09.949670 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 22:52:09.950128 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:09.950097 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zqssw" Nov 02 22:52:09.994190 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:09.994159 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cs2mc" event={"ID":"668094b0-0e10-47aa-ba49-6944f7083bc6","Type":"ContainerStarted","Data":"2b04a86bbd86730d43e2346f5cc1a7524a63cb881311d72e1228a94367d7e69f"} Nov 02 22:52:10.184710 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:10.184607 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cs2mc" podStartSLOduration=7.28301169 podStartE2EDuration="21.184593037s" totalImagesPullingTime="13.901581347s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:51:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:10.184320109 +0000 UTC m=+21.858896910" watchObservedRunningTime="2026-11-02 22:52:10.184593037 +0000 UTC m=+21.859169836" Nov 02 22:52:10.999995 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:10.999959 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" event={"ID":"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49","Type":"ContainerStarted","Data":"43223de84edfe26793fa484a6ad5d7c412b0a3cfb86222520067bc8ab6736869"} Nov 02 22:52:11.228200 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:11.227943 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" podStartSLOduration=7.282255047 podStartE2EDuration="22.227929419s" totalImagesPullingTime="14.945674372s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:51:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:11.176001778 +0000 UTC m=+22.850578579" watchObservedRunningTime="2026-11-02 22:52:11.227929419 +0000 UTC m=+22.902506219" Nov 02 22:52:11.851431 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:11.851400 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rmgpt" Nov 02 22:52:11.851431 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:11.851433 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:52:11.851626 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:11.851503 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rmgpt" podUID="e3a3bc57-6813-441c-bdc0-d39eb8214c61" Nov 02 22:52:11.851626 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:11.851401 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zqrz" Nov 02 22:52:11.851626 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:11.851616 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4gfs" podUID="fd96552a-1a78-4fac-8faf-71d36f010905" Nov 02 22:52:11.851713 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:11.851695 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zqrz" podUID="fc69f07f-4001-4951-aff8-d3d6671cc5d4" Nov 02 22:52:12.002972 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:12.002943 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:12.003428 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:12.003009 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:12.003428 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:12.003041 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:12.018729 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:12.018700 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:12.018917 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:12.018905 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:13.577831 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:13.577801 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-k4gfs"] Nov 02 22:52:13.578396 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:13.577937 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:52:13.578396 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:13.578023 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4gfs" podUID="fd96552a-1a78-4fac-8faf-71d36f010905" Nov 02 22:52:13.580704 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:13.580687 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9zqrz"] Nov 02 22:52:13.580788 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:13.580764 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zqrz" Nov 02 22:52:13.580853 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:13.580837 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zqrz" podUID="fc69f07f-4001-4951-aff8-d3d6671cc5d4" Nov 02 22:52:13.629507 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:13.629474 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rmgpt"] Nov 02 22:52:13.629651 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:13.629578 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rmgpt" Nov 02 22:52:13.629713 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:13.629673 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rmgpt" podUID="e3a3bc57-6813-441c-bdc0-d39eb8214c61" Nov 02 22:52:14.850954 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:14.850922 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:52:14.851423 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:14.851036 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4gfs" podUID="fd96552a-1a78-4fac-8faf-71d36f010905" Nov 02 22:52:15.851593 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:15.851517 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zqrz" Nov 02 22:52:15.851593 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:15.851546 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rmgpt" Nov 02 22:52:15.852134 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:15.851629 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zqrz" podUID="fc69f07f-4001-4951-aff8-d3d6671cc5d4" Nov 02 22:52:15.852134 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:15.851734 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rmgpt" podUID="e3a3bc57-6813-441c-bdc0-d39eb8214c61" Nov 02 22:52:16.850984 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:16.850954 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:52:16.851150 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:16.851056 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4gfs" podUID="fd96552a-1a78-4fac-8faf-71d36f010905" Nov 02 22:52:17.851183 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:17.851152 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zqrz" Nov 02 22:52:17.851555 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:17.851260 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zqrz" podUID="fc69f07f-4001-4951-aff8-d3d6671cc5d4" Nov 02 22:52:17.851555 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:17.851326 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rmgpt" Nov 02 22:52:17.851555 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:17.851446 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rmgpt" podUID="e3a3bc57-6813-441c-bdc0-d39eb8214c61" Nov 02 22:52:18.851490 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:18.851456 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:52:18.851852 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:18.851554 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4gfs" podUID="fd96552a-1a78-4fac-8faf-71d36f010905" Nov 02 22:52:19.679924 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.679718 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-211.us-west-2.compute.internal" event="NodeReady" Nov 02 22:52:19.680083 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.680016 2575 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 22:52:19.845161 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.845085 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fdj8t"] Nov 02 22:52:19.846729 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.846703 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-mtrcx"] Nov 02 22:52:19.846856 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.846753 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-fdj8t" Nov 02 22:52:19.849016 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.848996 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-pxpxp"] Nov 02 22:52:19.849174 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.849159 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-mtrcx" Nov 02 22:52:19.850467 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.850453 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-mbmf9"] Nov 02 22:52:19.850578 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.850563 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-pxpxp" Nov 02 22:52:19.851757 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.851740 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-896ccfb6c-q4rg5"] Nov 02 22:52:19.852129 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.851845 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbmf9" Nov 02 22:52:19.853149 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.853132 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ht25s"] Nov 02 22:52:19.853256 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.853240 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rmgpt" Nov 02 22:52:19.853305 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.853269 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:19.853619 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.853482 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zqrz" Nov 02 22:52:19.855067 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.855044 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cgpkk"] Nov 02 22:52:19.855324 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.855308 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ht25s" Nov 02 22:52:19.856565 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.856548 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-867x9"] Nov 02 22:52:19.856656 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.856599 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cgpkk" Nov 02 22:52:19.858351 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.858333 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-p66wt"] Nov 02 22:52:19.858438 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.858428 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-867x9" Nov 02 22:52:19.863201 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.863182 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-c2gxc"] Nov 02 22:52:19.863319 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.863305 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p66wt" Nov 02 22:52:19.865000 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.864984 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7b5b8cd649-z4rbs"] Nov 02 22:52:19.865091 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.865078 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-c2gxc" Nov 02 22:52:19.866425 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.866411 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:19.897110 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.897085 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-kkzvx"] Nov 02 22:52:19.898874 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.898852 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-777f69544d-d75n6"] Nov 02 22:52:19.899013 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.898994 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-kkzvx" Nov 02 22:52:19.900701 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.900679 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 02 22:52:19.900797 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.900740 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 02 22:52:19.900852 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.900818 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-xszmw"] Nov 02 22:52:19.901011 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.900995 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:19.904213 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.904197 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-6zkx2"] Nov 02 22:52:19.904340 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.904324 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-xszmw" Nov 02 22:52:19.905706 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.905687 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 02 22:52:19.906658 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.906641 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2rjj4\"" Nov 02 22:52:19.906763 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.906706 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-h9lck"] Nov 02 22:52:19.906822 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.906799 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6zkx2" Nov 02 22:52:19.908414 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.908398 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pvc5f"] Nov 02 22:52:19.908565 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.908511 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h9lck" Nov 02 22:52:19.910268 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.910252 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fdj8t"] Nov 02 22:52:19.910407 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.910277 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-mtrcx"] Nov 02 22:52:19.910407 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.910290 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-pxpxp"] Nov 02 22:52:19.910407 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.910306 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-mbmf9"] Nov 02 22:52:19.910407 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.910360 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pvc5f" Nov 02 22:52:19.910868 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.910722 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5kxql\"" Nov 02 22:52:19.910868 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.910820 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 02 22:52:19.910981 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.910872 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 02 22:52:19.910981 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.910874 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 22:52:19.910981 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.910952 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 02 22:52:19.910981 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.910974 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 02 22:52:19.916762 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.916742 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-p66wt"] Nov 02 22:52:19.920498 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.920480 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ht25s"] Nov 02 22:52:19.921494 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.921476 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cgpkk"] Nov 02 22:52:19.949002 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.948977 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-896ccfb6c-q4rg5"] Nov 02 22:52:19.950482 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.950453 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-kkzvx"] Nov 02 22:52:19.956862 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.956583 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pvc5f"] Nov 02 22:52:19.957690 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.957638 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncmnm\" (UniqueName: \"kubernetes.io/projected/4b9cd8c7-228b-451a-afe4-3af7a3bf0a42-kube-api-access-ncmnm\") pod \"kas-connection-checker-685895c7c9-mtrcx\" (UID: \"4b9cd8c7-228b-451a-afe4-3af7a3bf0a42\") " pod="kube-system/kas-connection-checker-685895c7c9-mtrcx" Nov 02 22:52:19.957690 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.957677 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/46eae5da-cf0b-4663-bdeb-f8220fa08c05-registry-certificates\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:19.957860 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.957709 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/df418d6f-b9d3-4019-967c-9511be123e1f-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ht25s\" (UID: \"df418d6f-b9d3-4019-967c-9511be123e1f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ht25s" Nov 02 22:52:19.957860 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.957734 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-default-certificate\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:19.957860 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.957759 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b858466f-3dcd-419c-9495-4dbb89e8d51b-service-ca-bundle\") pod \"insights-operator-7c4986d449-xszmw\" (UID: \"b858466f-3dcd-419c-9495-4dbb89e8d51b\") " pod="openshift-insights/insights-operator-7c4986d449-xszmw" Nov 02 22:52:19.957860 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.957785 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9c4b6fbd-74cf-41c6-bbae-92748ac25e1e-config\") pod \"console-operator-7cd56b8dbc-c2gxc\" (UID: \"9c4b6fbd-74cf-41c6-bbae-92748ac25e1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c2gxc" Nov 02 22:52:19.958602 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.957878 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/105406c8-6800-4562-9651-962bee9f8d73-config-volume\") pod \"dns-default-pvc5f\" (UID: \"105406c8-6800-4562-9651-962bee9f8d73\") " pod="openshift-dns/dns-default-pvc5f" Nov 02 22:52:19.958602 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.957924 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-metrics-certs\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:19.958602 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.957941 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/46eae5da-cf0b-4663-bdeb-f8220fa08c05-trusted-ca\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:19.958602 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.957959 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/df418d6f-b9d3-4019-967c-9511be123e1f-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ht25s\" (UID: \"df418d6f-b9d3-4019-967c-9511be123e1f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ht25s" Nov 02 22:52:19.958602 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.957980 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/60832943-bf1f-4685-be00-08152c423b08-config\") pod \"service-ca-operator-544d749c8c-p66wt\" (UID: \"60832943-bf1f-4685-be00-08152c423b08\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p66wt" Nov 02 22:52:19.958602 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.958166 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnt9q\" (UniqueName: \"kubernetes.io/projected/15789a0d-e030-45e0-a129-41de738ef767-kube-api-access-jnt9q\") pod \"cluster-monitoring-operator-884576c57-mbmf9\" (UID: \"15789a0d-e030-45e0-a129-41de738ef767\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbmf9" Nov 02 22:52:19.958602 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.958213 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89zw7\" (UniqueName: \"kubernetes.io/projected/9c4b6fbd-74cf-41c6-bbae-92748ac25e1e-kube-api-access-89zw7\") pod \"console-operator-7cd56b8dbc-c2gxc\" (UID: \"9c4b6fbd-74cf-41c6-bbae-92748ac25e1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c2gxc" Nov 02 22:52:19.958602 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.958266 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/105406c8-6800-4562-9651-962bee9f8d73-tmp-dir\") pod \"dns-default-pvc5f\" (UID: \"105406c8-6800-4562-9651-962bee9f8d73\") " pod="openshift-dns/dns-default-pvc5f" Nov 02 22:52:19.958602 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.958296 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwnxj\" (UniqueName: \"kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-kube-api-access-nwnxj\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:19.958602 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.958323 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-registry-tls\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:19.958602 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.958345 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-bound-sa-token\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:19.958602 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.958373 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/00b95c29-7df8-4602-bd0b-06b6281d5dc1-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-cgpkk\" (UID: \"00b95c29-7df8-4602-bd0b-06b6281d5dc1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cgpkk" Nov 02 22:52:19.958602 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.958397 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0237eeb1-8c85-475e-8580-6dec324de705-registry-certificates\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:19.958602 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.958419 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b858466f-3dcd-419c-9495-4dbb89e8d51b-tmp\") pod \"insights-operator-7c4986d449-xszmw\" (UID: \"b858466f-3dcd-419c-9495-4dbb89e8d51b\") " pod="openshift-insights/insights-operator-7c4986d449-xszmw" Nov 02 22:52:19.958602 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.958447 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxxqc\" (UniqueName: \"kubernetes.io/projected/236450f1-d9fa-488b-9022-ab4f5517549f-kube-api-access-vxxqc\") pod \"network-check-source-6f78d59f99-6zkx2\" (UID: \"236450f1-d9fa-488b-9022-ab4f5517549f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6zkx2" Nov 02 22:52:19.959274 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.958502 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-registry-tls\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:19.959274 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.958528 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t8ms\" (UniqueName: \"kubernetes.io/projected/ac4a7fda-863e-4fb1-a98c-9b325a8c0df7-kube-api-access-9t8ms\") pod \"ingress-canary-h9lck\" (UID: \"ac4a7fda-863e-4fb1-a98c-9b325a8c0df7\") " pod="openshift-ingress-canary/ingress-canary-h9lck" Nov 02 22:52:19.959274 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.958551 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6c7074b-4caa-43fb-9fca-14ebb5abf958-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kkzvx\" (UID: \"b6c7074b-4caa-43fb-9fca-14ebb5abf958\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkzvx" Nov 02 22:52:19.959274 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.958578 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt847\" (UniqueName: \"kubernetes.io/projected/00b95c29-7df8-4602-bd0b-06b6281d5dc1-kube-api-access-zt847\") pod \"cluster-samples-operator-6ddb6c8b94-cgpkk\" (UID: \"00b95c29-7df8-4602-bd0b-06b6281d5dc1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cgpkk" Nov 02 22:52:19.959274 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.958601 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md7ml\" (UniqueName: \"kubernetes.io/projected/75f7d1d7-88e4-41e0-bdd3-a43df363b408-kube-api-access-md7ml\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:19.959274 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.958627 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/15789a0d-e030-45e0-a129-41de738ef767-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-mbmf9\" (UID: \"15789a0d-e030-45e0-a129-41de738ef767\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbmf9" Nov 02 22:52:19.959274 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.958654 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0237eeb1-8c85-475e-8580-6dec324de705-image-registry-private-configuration\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:19.959274 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.958700 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkzvb\" (UniqueName: \"kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-kube-api-access-nkzvb\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:19.959274 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.958724 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0237eeb1-8c85-475e-8580-6dec324de705-installation-pull-secrets\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:19.959274 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.958780 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/60832943-bf1f-4685-be00-08152c423b08-serving-cert\") pod \"service-ca-operator-544d749c8c-p66wt\" (UID: \"60832943-bf1f-4685-be00-08152c423b08\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p66wt" Nov 02 22:52:19.959274 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.958807 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f7d1d7-88e4-41e0-bdd3-a43df363b408-service-ca-bundle\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:19.959274 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.958831 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmk5v\" (UniqueName: \"kubernetes.io/projected/60832943-bf1f-4685-be00-08152c423b08-kube-api-access-pmk5v\") pod \"service-ca-operator-544d749c8c-p66wt\" (UID: \"60832943-bf1f-4685-be00-08152c423b08\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p66wt" Nov 02 22:52:19.959274 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.958854 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/46eae5da-cf0b-4663-bdeb-f8220fa08c05-ca-trust-extracted\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:19.959274 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.958874 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac4a7fda-863e-4fb1-a98c-9b325a8c0df7-cert\") pod \"ingress-canary-h9lck\" (UID: \"ac4a7fda-863e-4fb1-a98c-9b325a8c0df7\") " pod="openshift-ingress-canary/ingress-canary-h9lck" Nov 02 22:52:19.959274 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.958921 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stv25\" (UniqueName: \"kubernetes.io/projected/42d598be-a6f1-486e-9ac3-2f5ece575f89-kube-api-access-stv25\") pod \"kas-connection-checker-685895c7c9-fdj8t\" (UID: \"42d598be-a6f1-486e-9ac3-2f5ece575f89\") " pod="kube-system/kas-connection-checker-685895c7c9-fdj8t" Nov 02 22:52:19.959756 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.958949 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b6c7074b-4caa-43fb-9fca-14ebb5abf958-nginx-conf\") pod \"networking-console-plugin-86c478bb88-kkzvx\" (UID: \"b6c7074b-4caa-43fb-9fca-14ebb5abf958\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkzvx" Nov 02 22:52:19.959756 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.958972 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b858466f-3dcd-419c-9495-4dbb89e8d51b-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xszmw\" (UID: \"b858466f-3dcd-419c-9495-4dbb89e8d51b\") " pod="openshift-insights/insights-operator-7c4986d449-xszmw" Nov 02 22:52:19.959756 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.959020 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9c4b6fbd-74cf-41c6-bbae-92748ac25e1e-serving-cert\") pod \"console-operator-7cd56b8dbc-c2gxc\" (UID: \"9c4b6fbd-74cf-41c6-bbae-92748ac25e1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c2gxc" Nov 02 22:52:19.959756 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.959049 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjljw\" (UniqueName: \"kubernetes.io/projected/df418d6f-b9d3-4019-967c-9511be123e1f-kube-api-access-jjljw\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ht25s\" (UID: \"df418d6f-b9d3-4019-967c-9511be123e1f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ht25s" Nov 02 22:52:19.959756 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.959097 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-bound-sa-token\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:19.959756 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.959130 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24smc\" (UniqueName: \"kubernetes.io/projected/fbcab4d0-9dd6-4682-be01-c4f589697e95-kube-api-access-24smc\") pod \"kas-connection-checker-685895c7c9-pxpxp\" (UID: \"fbcab4d0-9dd6-4682-be01-c4f589697e95\") " pod="kube-system/kas-connection-checker-685895c7c9-pxpxp" Nov 02 22:52:19.959756 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.959178 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-867x9"] Nov 02 22:52:19.959756 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.959179 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/105406c8-6800-4562-9651-962bee9f8d73-metrics-tls\") pod \"dns-default-pvc5f\" (UID: \"105406c8-6800-4562-9651-962bee9f8d73\") " pod="openshift-dns/dns-default-pvc5f" Nov 02 22:52:19.959756 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.959254 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/46eae5da-cf0b-4663-bdeb-f8220fa08c05-image-registry-private-configuration\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:19.959756 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.959291 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0237eeb1-8c85-475e-8580-6dec324de705-trusted-ca\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:19.959756 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.959316 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b858466f-3dcd-419c-9495-4dbb89e8d51b-serving-cert\") pod \"insights-operator-7c4986d449-xszmw\" (UID: \"b858466f-3dcd-419c-9495-4dbb89e8d51b\") " pod="openshift-insights/insights-operator-7c4986d449-xszmw" Nov 02 22:52:19.959756 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.959354 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-stats-auth\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:19.959756 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.959382 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9c4b6fbd-74cf-41c6-bbae-92748ac25e1e-trusted-ca\") pod \"console-operator-7cd56b8dbc-c2gxc\" (UID: \"9c4b6fbd-74cf-41c6-bbae-92748ac25e1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c2gxc" Nov 02 22:52:19.959756 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.959415 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/15789a0d-e030-45e0-a129-41de738ef767-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mbmf9\" (UID: \"15789a0d-e030-45e0-a129-41de738ef767\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbmf9" Nov 02 22:52:19.959756 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.959454 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/46eae5da-cf0b-4663-bdeb-f8220fa08c05-installation-pull-secrets\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:19.960341 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.959490 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws5j7\" (UniqueName: \"kubernetes.io/projected/b858466f-3dcd-419c-9495-4dbb89e8d51b-kube-api-access-ws5j7\") pod \"insights-operator-7c4986d449-xszmw\" (UID: \"b858466f-3dcd-419c-9495-4dbb89e8d51b\") " pod="openshift-insights/insights-operator-7c4986d449-xszmw" Nov 02 22:52:19.960341 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.959517 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/00b95c29-7df8-4602-bd0b-06b6281d5dc1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-cgpkk\" (UID: \"00b95c29-7df8-4602-bd0b-06b6281d5dc1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cgpkk" Nov 02 22:52:19.960341 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.959542 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0237eeb1-8c85-475e-8580-6dec324de705-ca-trust-extracted\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:19.960341 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.959571 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqdk5\" (UniqueName: \"kubernetes.io/projected/0d008a27-d5ba-49cb-b3fb-abdecd723abd-kube-api-access-fqdk5\") pod \"volume-data-source-validator-58c9f5f46d-867x9\" (UID: \"0d008a27-d5ba-49cb-b3fb-abdecd723abd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-867x9" Nov 02 22:52:19.960341 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.959590 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b858466f-3dcd-419c-9495-4dbb89e8d51b-snapshots\") pod \"insights-operator-7c4986d449-xszmw\" (UID: \"b858466f-3dcd-419c-9495-4dbb89e8d51b\") " pod="openshift-insights/insights-operator-7c4986d449-xszmw" Nov 02 22:52:19.960341 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.959608 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f4r4\" (UniqueName: \"kubernetes.io/projected/105406c8-6800-4562-9651-962bee9f8d73-kube-api-access-9f4r4\") pod \"dns-default-pvc5f\" (UID: \"105406c8-6800-4562-9651-962bee9f8d73\") " pod="openshift-dns/dns-default-pvc5f" Nov 02 22:52:19.963623 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.963539 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b5b8cd649-z4rbs"] Nov 02 22:52:19.991310 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.991286 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-xszmw"] Nov 02 22:52:19.992086 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.992069 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-6zkx2"] Nov 02 22:52:19.994998 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.994978 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h9lck"] Nov 02 22:52:19.995924 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:19.995883 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-c2gxc"] Nov 02 22:52:20.001001 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.000972 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 02 22:52:20.001001 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.000977 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:52:20.001240 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.001008 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 02 22:52:20.001596 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.001549 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 02 22:52:20.001596 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.001588 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 02 22:52:20.001802 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.001662 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 02 22:52:20.001802 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.001790 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:52:20.001974 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.001951 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 22:52:20.002135 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.002116 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 02 22:52:20.002257 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.002155 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:52:20.002347 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.002172 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 22:52:20.002432 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.002198 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 22:52:20.002432 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.002299 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 02 22:52:20.002597 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.002440 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 22:52:20.002597 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.002470 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 02 22:52:20.002597 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.002479 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 02 22:52:20.002597 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.002486 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:52:20.002597 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.002368 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 22:52:20.002597 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.002388 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 22:52:20.002597 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.002416 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 02 22:52:20.002597 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.002516 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 22:52:20.002597 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.002424 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:52:20.002597 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.002344 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 22:52:20.002597 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.002398 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 22:52:20.002597 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.002374 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 02 22:52:20.002597 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.002518 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 22:52:20.002597 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.002557 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 02 22:52:20.002597 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.002563 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 22:52:20.053238 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.053206 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 02 22:52:20.054479 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.054451 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 02 22:52:20.054847 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.054749 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 02 22:52:20.054847 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.054774 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 02 22:52:20.054847 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.054783 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 02 22:52:20.054847 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.054755 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 02 22:52:20.054847 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.054800 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 02 22:52:20.054847 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.054819 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 22:52:20.054847 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.054820 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 02 22:52:20.054847 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.054820 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 22:52:20.054847 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.054843 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 02 22:52:20.055348 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.054916 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 02 22:52:20.055348 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.054970 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-r5gxd\"" Nov 02 22:52:20.055348 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.054984 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 02 22:52:20.055348 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.055023 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t6gjv\"" Nov 02 22:52:20.055348 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.055075 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-8zgv4\"" Nov 02 22:52:20.055348 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.055178 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 02 22:52:20.055725 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.055709 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-777f69544d-d75n6"] Nov 02 22:52:20.060183 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.060167 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0237eeb1-8c85-475e-8580-6dec324de705-trusted-ca\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:20.060225 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.060191 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b858466f-3dcd-419c-9495-4dbb89e8d51b-serving-cert\") pod \"insights-operator-7c4986d449-xszmw\" (UID: \"b858466f-3dcd-419c-9495-4dbb89e8d51b\") " pod="openshift-insights/insights-operator-7c4986d449-xszmw" Nov 02 22:52:20.060225 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.060218 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-stats-auth\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:20.060323 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.060233 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9c4b6fbd-74cf-41c6-bbae-92748ac25e1e-trusted-ca\") pod \"console-operator-7cd56b8dbc-c2gxc\" (UID: \"9c4b6fbd-74cf-41c6-bbae-92748ac25e1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c2gxc" Nov 02 22:52:20.060323 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.060258 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/15789a0d-e030-45e0-a129-41de738ef767-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mbmf9\" (UID: \"15789a0d-e030-45e0-a129-41de738ef767\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbmf9" Nov 02 22:52:20.060323 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.060286 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/46eae5da-cf0b-4663-bdeb-f8220fa08c05-installation-pull-secrets\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:20.060323 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.060318 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws5j7\" (UniqueName: \"kubernetes.io/projected/b858466f-3dcd-419c-9495-4dbb89e8d51b-kube-api-access-ws5j7\") pod \"insights-operator-7c4986d449-xszmw\" (UID: \"b858466f-3dcd-419c-9495-4dbb89e8d51b\") " pod="openshift-insights/insights-operator-7c4986d449-xszmw" Nov 02 22:52:20.060496 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.060341 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/00b95c29-7df8-4602-bd0b-06b6281d5dc1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-cgpkk\" (UID: \"00b95c29-7df8-4602-bd0b-06b6281d5dc1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cgpkk" Nov 02 22:52:20.060496 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.060368 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0237eeb1-8c85-475e-8580-6dec324de705-ca-trust-extracted\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:20.060932 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:20.060377 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 22:52:20.061035 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.060394 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqdk5\" (UniqueName: \"kubernetes.io/projected/0d008a27-d5ba-49cb-b3fb-abdecd723abd-kube-api-access-fqdk5\") pod \"volume-data-source-validator-58c9f5f46d-867x9\" (UID: \"0d008a27-d5ba-49cb-b3fb-abdecd723abd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-867x9" Nov 02 22:52:20.061035 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.060977 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b858466f-3dcd-419c-9495-4dbb89e8d51b-snapshots\") pod \"insights-operator-7c4986d449-xszmw\" (UID: \"b858466f-3dcd-419c-9495-4dbb89e8d51b\") " pod="openshift-insights/insights-operator-7c4986d449-xszmw" Nov 02 22:52:20.061035 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:20.060991 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15789a0d-e030-45e0-a129-41de738ef767-cluster-monitoring-operator-tls podName:15789a0d-e030-45e0-a129-41de738ef767 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:20.560970424 +0000 UTC m=+32.235547213 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/15789a0d-e030-45e0-a129-41de738ef767-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-mbmf9" (UID: "15789a0d-e030-45e0-a129-41de738ef767") : secret "cluster-monitoring-operator-tls" not found Nov 02 22:52:20.061035 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.061024 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f4r4\" (UniqueName: \"kubernetes.io/projected/105406c8-6800-4562-9651-962bee9f8d73-kube-api-access-9f4r4\") pod \"dns-default-pvc5f\" (UID: \"105406c8-6800-4562-9651-962bee9f8d73\") " pod="openshift-dns/dns-default-pvc5f" Nov 02 22:52:20.061203 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.061051 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncmnm\" (UniqueName: \"kubernetes.io/projected/4b9cd8c7-228b-451a-afe4-3af7a3bf0a42-kube-api-access-ncmnm\") pod \"kas-connection-checker-685895c7c9-mtrcx\" (UID: \"4b9cd8c7-228b-451a-afe4-3af7a3bf0a42\") " pod="kube-system/kas-connection-checker-685895c7c9-mtrcx" Nov 02 22:52:20.061203 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.061076 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/46eae5da-cf0b-4663-bdeb-f8220fa08c05-registry-certificates\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:20.061203 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.061104 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/df418d6f-b9d3-4019-967c-9511be123e1f-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ht25s\" (UID: \"df418d6f-b9d3-4019-967c-9511be123e1f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ht25s" Nov 02 22:52:20.061203 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.061125 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-default-certificate\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:20.061203 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.061146 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b858466f-3dcd-419c-9495-4dbb89e8d51b-service-ca-bundle\") pod \"insights-operator-7c4986d449-xszmw\" (UID: \"b858466f-3dcd-419c-9495-4dbb89e8d51b\") " pod="openshift-insights/insights-operator-7c4986d449-xszmw" Nov 02 22:52:20.061203 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.061167 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9c4b6fbd-74cf-41c6-bbae-92748ac25e1e-config\") pod \"console-operator-7cd56b8dbc-c2gxc\" (UID: \"9c4b6fbd-74cf-41c6-bbae-92748ac25e1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c2gxc" Nov 02 22:52:20.061203 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.061194 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/105406c8-6800-4562-9651-962bee9f8d73-config-volume\") pod \"dns-default-pvc5f\" (UID: \"105406c8-6800-4562-9651-962bee9f8d73\") " pod="openshift-dns/dns-default-pvc5f" Nov 02 22:52:20.061479 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.061219 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-metrics-certs\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:20.061479 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.061244 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/46eae5da-cf0b-4663-bdeb-f8220fa08c05-trusted-ca\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:20.061479 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.061272 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/df418d6f-b9d3-4019-967c-9511be123e1f-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ht25s\" (UID: \"df418d6f-b9d3-4019-967c-9511be123e1f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ht25s" Nov 02 22:52:20.061479 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.061302 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/60832943-bf1f-4685-be00-08152c423b08-config\") pod \"service-ca-operator-544d749c8c-p66wt\" (UID: \"60832943-bf1f-4685-be00-08152c423b08\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p66wt" Nov 02 22:52:20.061479 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.061329 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnt9q\" (UniqueName: \"kubernetes.io/projected/15789a0d-e030-45e0-a129-41de738ef767-kube-api-access-jnt9q\") pod \"cluster-monitoring-operator-884576c57-mbmf9\" (UID: \"15789a0d-e030-45e0-a129-41de738ef767\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbmf9" Nov 02 22:52:20.061479 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.061350 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0237eeb1-8c85-475e-8580-6dec324de705-trusted-ca\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:20.061479 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.061353 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89zw7\" (UniqueName: \"kubernetes.io/projected/9c4b6fbd-74cf-41c6-bbae-92748ac25e1e-kube-api-access-89zw7\") pod \"console-operator-7cd56b8dbc-c2gxc\" (UID: \"9c4b6fbd-74cf-41c6-bbae-92748ac25e1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c2gxc" Nov 02 22:52:20.061479 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.061383 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9c4b6fbd-74cf-41c6-bbae-92748ac25e1e-trusted-ca\") pod \"console-operator-7cd56b8dbc-c2gxc\" (UID: \"9c4b6fbd-74cf-41c6-bbae-92748ac25e1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c2gxc" Nov 02 22:52:20.061479 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.060872 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0237eeb1-8c85-475e-8580-6dec324de705-ca-trust-extracted\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:20.061479 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.061412 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/105406c8-6800-4562-9651-962bee9f8d73-tmp-dir\") pod \"dns-default-pvc5f\" (UID: \"105406c8-6800-4562-9651-962bee9f8d73\") " pod="openshift-dns/dns-default-pvc5f" Nov 02 22:52:20.061479 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.061443 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwnxj\" (UniqueName: \"kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-kube-api-access-nwnxj\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:20.061479 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.061466 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-registry-tls\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:20.061956 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.061489 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-bound-sa-token\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:20.061956 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.061514 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/00b95c29-7df8-4602-bd0b-06b6281d5dc1-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-cgpkk\" (UID: \"00b95c29-7df8-4602-bd0b-06b6281d5dc1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cgpkk" Nov 02 22:52:20.061956 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.061515 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b858466f-3dcd-419c-9495-4dbb89e8d51b-snapshots\") pod \"insights-operator-7c4986d449-xszmw\" (UID: \"b858466f-3dcd-419c-9495-4dbb89e8d51b\") " pod="openshift-insights/insights-operator-7c4986d449-xszmw" Nov 02 22:52:20.061956 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.061535 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0237eeb1-8c85-475e-8580-6dec324de705-registry-certificates\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:20.061956 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.061555 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b858466f-3dcd-419c-9495-4dbb89e8d51b-tmp\") pod \"insights-operator-7c4986d449-xszmw\" (UID: \"b858466f-3dcd-419c-9495-4dbb89e8d51b\") " pod="openshift-insights/insights-operator-7c4986d449-xszmw" Nov 02 22:52:20.061956 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.061582 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxxqc\" (UniqueName: \"kubernetes.io/projected/236450f1-d9fa-488b-9022-ab4f5517549f-kube-api-access-vxxqc\") pod \"network-check-source-6f78d59f99-6zkx2\" (UID: \"236450f1-d9fa-488b-9022-ab4f5517549f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6zkx2" Nov 02 22:52:20.061956 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.061602 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-registry-tls\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:20.061956 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.061623 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t8ms\" (UniqueName: \"kubernetes.io/projected/ac4a7fda-863e-4fb1-a98c-9b325a8c0df7-kube-api-access-9t8ms\") pod \"ingress-canary-h9lck\" (UID: \"ac4a7fda-863e-4fb1-a98c-9b325a8c0df7\") " pod="openshift-ingress-canary/ingress-canary-h9lck" Nov 02 22:52:20.061956 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.061646 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6c7074b-4caa-43fb-9fca-14ebb5abf958-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kkzvx\" (UID: \"b6c7074b-4caa-43fb-9fca-14ebb5abf958\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkzvx" Nov 02 22:52:20.061956 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.061671 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt847\" (UniqueName: \"kubernetes.io/projected/00b95c29-7df8-4602-bd0b-06b6281d5dc1-kube-api-access-zt847\") pod \"cluster-samples-operator-6ddb6c8b94-cgpkk\" (UID: \"00b95c29-7df8-4602-bd0b-06b6281d5dc1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cgpkk" Nov 02 22:52:20.061956 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.061694 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md7ml\" (UniqueName: \"kubernetes.io/projected/75f7d1d7-88e4-41e0-bdd3-a43df363b408-kube-api-access-md7ml\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:20.061956 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:20.060599 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 22:52:20.061956 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.061715 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/15789a0d-e030-45e0-a129-41de738ef767-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-mbmf9\" (UID: \"15789a0d-e030-45e0-a129-41de738ef767\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbmf9" Nov 02 22:52:20.061956 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.061738 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0237eeb1-8c85-475e-8580-6dec324de705-image-registry-private-configuration\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:20.061956 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.061763 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkzvb\" (UniqueName: \"kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-kube-api-access-nkzvb\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:20.062602 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:20.061781 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00b95c29-7df8-4602-bd0b-06b6281d5dc1-samples-operator-tls podName:00b95c29-7df8-4602-bd0b-06b6281d5dc1 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:20.561764317 +0000 UTC m=+32.236341111 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/00b95c29-7df8-4602-bd0b-06b6281d5dc1-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-cgpkk" (UID: "00b95c29-7df8-4602-bd0b-06b6281d5dc1") : secret "samples-operator-tls" not found Nov 02 22:52:20.062602 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.061816 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0237eeb1-8c85-475e-8580-6dec324de705-installation-pull-secrets\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:20.062602 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.061853 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b858466f-3dcd-419c-9495-4dbb89e8d51b-service-ca-bundle\") pod \"insights-operator-7c4986d449-xszmw\" (UID: \"b858466f-3dcd-419c-9495-4dbb89e8d51b\") " pod="openshift-insights/insights-operator-7c4986d449-xszmw" Nov 02 22:52:20.062602 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.061916 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/60832943-bf1f-4685-be00-08152c423b08-serving-cert\") pod \"service-ca-operator-544d749c8c-p66wt\" (UID: \"60832943-bf1f-4685-be00-08152c423b08\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p66wt" Nov 02 22:52:20.062602 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.061950 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f7d1d7-88e4-41e0-bdd3-a43df363b408-service-ca-bundle\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:20.062602 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.061981 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmk5v\" (UniqueName: \"kubernetes.io/projected/60832943-bf1f-4685-be00-08152c423b08-kube-api-access-pmk5v\") pod \"service-ca-operator-544d749c8c-p66wt\" (UID: \"60832943-bf1f-4685-be00-08152c423b08\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p66wt" Nov 02 22:52:20.062602 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.062005 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/46eae5da-cf0b-4663-bdeb-f8220fa08c05-ca-trust-extracted\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:20.062602 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.062030 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac4a7fda-863e-4fb1-a98c-9b325a8c0df7-cert\") pod \"ingress-canary-h9lck\" (UID: \"ac4a7fda-863e-4fb1-a98c-9b325a8c0df7\") " pod="openshift-ingress-canary/ingress-canary-h9lck" Nov 02 22:52:20.062602 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.062056 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stv25\" (UniqueName: \"kubernetes.io/projected/42d598be-a6f1-486e-9ac3-2f5ece575f89-kube-api-access-stv25\") pod \"kas-connection-checker-685895c7c9-fdj8t\" (UID: \"42d598be-a6f1-486e-9ac3-2f5ece575f89\") " pod="kube-system/kas-connection-checker-685895c7c9-fdj8t" Nov 02 22:52:20.062602 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.062075 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/105406c8-6800-4562-9651-962bee9f8d73-config-volume\") pod \"dns-default-pvc5f\" (UID: \"105406c8-6800-4562-9651-962bee9f8d73\") " pod="openshift-dns/dns-default-pvc5f" Nov 02 22:52:20.062602 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:20.062165 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 22:52:20.062602 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:20.062210 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-metrics-certs podName:75f7d1d7-88e4-41e0-bdd3-a43df363b408 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:20.562194221 +0000 UTC m=+32.236771005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-metrics-certs") pod "router-default-896ccfb6c-q4rg5" (UID: "75f7d1d7-88e4-41e0-bdd3-a43df363b408") : secret "router-metrics-certs-default" not found Nov 02 22:52:20.062602 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.062227 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/105406c8-6800-4562-9651-962bee9f8d73-tmp-dir\") pod \"dns-default-pvc5f\" (UID: \"105406c8-6800-4562-9651-962bee9f8d73\") " pod="openshift-dns/dns-default-pvc5f" Nov 02 22:52:20.063242 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:20.062622 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75f7d1d7-88e4-41e0-bdd3-a43df363b408-service-ca-bundle podName:75f7d1d7-88e4-41e0-bdd3-a43df363b408 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:20.562608248 +0000 UTC m=+32.237185048 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/75f7d1d7-88e4-41e0-bdd3-a43df363b408-service-ca-bundle") pod "router-default-896ccfb6c-q4rg5" (UID: "75f7d1d7-88e4-41e0-bdd3-a43df363b408") : configmap references non-existent config key: service-ca.crt Nov 02 22:52:20.063242 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.062620 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/46eae5da-cf0b-4663-bdeb-f8220fa08c05-registry-certificates\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:20.063242 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.062813 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/60832943-bf1f-4685-be00-08152c423b08-config\") pod \"service-ca-operator-544d749c8c-p66wt\" (UID: \"60832943-bf1f-4685-be00-08152c423b08\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p66wt" Nov 02 22:52:20.063242 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.062980 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9c4b6fbd-74cf-41c6-bbae-92748ac25e1e-config\") pod \"console-operator-7cd56b8dbc-c2gxc\" (UID: \"9c4b6fbd-74cf-41c6-bbae-92748ac25e1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c2gxc" Nov 02 22:52:20.063242 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.062081 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b6c7074b-4caa-43fb-9fca-14ebb5abf958-nginx-conf\") pod \"networking-console-plugin-86c478bb88-kkzvx\" (UID: \"b6c7074b-4caa-43fb-9fca-14ebb5abf958\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkzvx" Nov 02 22:52:20.063242 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.063071 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b858466f-3dcd-419c-9495-4dbb89e8d51b-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xszmw\" (UID: \"b858466f-3dcd-419c-9495-4dbb89e8d51b\") " pod="openshift-insights/insights-operator-7c4986d449-xszmw" Nov 02 22:52:20.063242 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.063103 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9c4b6fbd-74cf-41c6-bbae-92748ac25e1e-serving-cert\") pod \"console-operator-7cd56b8dbc-c2gxc\" (UID: \"9c4b6fbd-74cf-41c6-bbae-92748ac25e1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c2gxc" Nov 02 22:52:20.063242 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.063133 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjljw\" (UniqueName: \"kubernetes.io/projected/df418d6f-b9d3-4019-967c-9511be123e1f-kube-api-access-jjljw\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ht25s\" (UID: \"df418d6f-b9d3-4019-967c-9511be123e1f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ht25s" Nov 02 22:52:20.063242 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.063161 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-bound-sa-token\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:20.063242 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.063193 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24smc\" (UniqueName: \"kubernetes.io/projected/fbcab4d0-9dd6-4682-be01-c4f589697e95-kube-api-access-24smc\") pod \"kas-connection-checker-685895c7c9-pxpxp\" (UID: \"fbcab4d0-9dd6-4682-be01-c4f589697e95\") " pod="kube-system/kas-connection-checker-685895c7c9-pxpxp" Nov 02 22:52:20.063242 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.063240 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/105406c8-6800-4562-9651-962bee9f8d73-metrics-tls\") pod \"dns-default-pvc5f\" (UID: \"105406c8-6800-4562-9651-962bee9f8d73\") " pod="openshift-dns/dns-default-pvc5f" Nov 02 22:52:20.063767 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:20.063257 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 22:52:20.063767 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.063267 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/46eae5da-cf0b-4663-bdeb-f8220fa08c05-image-registry-private-configuration\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:20.063767 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:20.063272 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-777f69544d-d75n6: secret "image-registry-tls" not found Nov 02 22:52:20.063767 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:20.063318 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-registry-tls podName:0237eeb1-8c85-475e-8580-6dec324de705 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:20.563302581 +0000 UTC m=+32.237879376 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-registry-tls") pod "image-registry-777f69544d-d75n6" (UID: "0237eeb1-8c85-475e-8580-6dec324de705") : secret "image-registry-tls" not found Nov 02 22:52:20.063767 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.063357 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b6c7074b-4caa-43fb-9fca-14ebb5abf958-nginx-conf\") pod \"networking-console-plugin-86c478bb88-kkzvx\" (UID: \"b6c7074b-4caa-43fb-9fca-14ebb5abf958\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkzvx" Nov 02 22:52:20.063767 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:20.063571 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 22:52:20.063767 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:20.063584 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b5b8cd649-z4rbs: secret "image-registry-tls" not found Nov 02 22:52:20.063767 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:20.063619 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-registry-tls podName:46eae5da-cf0b-4663-bdeb-f8220fa08c05 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:20.563607148 +0000 UTC m=+32.238183961 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-registry-tls") pod "image-registry-7b5b8cd649-z4rbs" (UID: "46eae5da-cf0b-4663-bdeb-f8220fa08c05") : secret "image-registry-tls" not found Nov 02 22:52:20.064169 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.063921 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/46eae5da-cf0b-4663-bdeb-f8220fa08c05-ca-trust-extracted\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:20.065515 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.064499 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/00b95c29-7df8-4602-bd0b-06b6281d5dc1-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-cgpkk\" (UID: \"00b95c29-7df8-4602-bd0b-06b6281d5dc1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cgpkk" Nov 02 22:52:20.065515 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:20.064621 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 22:52:20.065515 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:20.064666 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6c7074b-4caa-43fb-9fca-14ebb5abf958-networking-console-plugin-cert podName:b6c7074b-4caa-43fb-9fca-14ebb5abf958 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:20.564651196 +0000 UTC m=+32.239227989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b6c7074b-4caa-43fb-9fca-14ebb5abf958-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kkzvx" (UID: "b6c7074b-4caa-43fb-9fca-14ebb5abf958") : secret "networking-console-plugin-cert" not found Nov 02 22:52:20.065515 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.063162 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/df418d6f-b9d3-4019-967c-9511be123e1f-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ht25s\" (UID: \"df418d6f-b9d3-4019-967c-9511be123e1f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ht25s" Nov 02 22:52:20.065515 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:20.064960 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 22:52:20.065515 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:20.065003 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/105406c8-6800-4562-9651-962bee9f8d73-metrics-tls podName:105406c8-6800-4562-9651-962bee9f8d73 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:20.564989574 +0000 UTC m=+32.239566355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/105406c8-6800-4562-9651-962bee9f8d73-metrics-tls") pod "dns-default-pvc5f" (UID: "105406c8-6800-4562-9651-962bee9f8d73") : secret "dns-default-metrics-tls" not found Nov 02 22:52:20.065515 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.065263 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b858466f-3dcd-419c-9495-4dbb89e8d51b-tmp\") pod \"insights-operator-7c4986d449-xszmw\" (UID: \"b858466f-3dcd-419c-9495-4dbb89e8d51b\") " pod="openshift-insights/insights-operator-7c4986d449-xszmw" Nov 02 22:52:20.065515 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.065468 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b858466f-3dcd-419c-9495-4dbb89e8d51b-serving-cert\") pod \"insights-operator-7c4986d449-xszmw\" (UID: \"b858466f-3dcd-419c-9495-4dbb89e8d51b\") " pod="openshift-insights/insights-operator-7c4986d449-xszmw" Nov 02 22:52:20.065981 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.065687 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/46eae5da-cf0b-4663-bdeb-f8220fa08c05-trusted-ca\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:20.065981 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.065757 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/df418d6f-b9d3-4019-967c-9511be123e1f-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ht25s\" (UID: \"df418d6f-b9d3-4019-967c-9511be123e1f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ht25s" Nov 02 22:52:20.066089 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.066012 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/46eae5da-cf0b-4663-bdeb-f8220fa08c05-installation-pull-secrets\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:20.066665 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.066617 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/15789a0d-e030-45e0-a129-41de738ef767-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-mbmf9\" (UID: \"15789a0d-e030-45e0-a129-41de738ef767\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbmf9" Nov 02 22:52:20.066851 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.066818 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0237eeb1-8c85-475e-8580-6dec324de705-registry-certificates\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:20.066950 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.066917 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-stats-auth\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:20.067248 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.067200 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b858466f-3dcd-419c-9495-4dbb89e8d51b-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xszmw\" (UID: \"b858466f-3dcd-419c-9495-4dbb89e8d51b\") " pod="openshift-insights/insights-operator-7c4986d449-xszmw" Nov 02 22:52:20.067513 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.067493 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac4a7fda-863e-4fb1-a98c-9b325a8c0df7-cert\") pod \"ingress-canary-h9lck\" (UID: \"ac4a7fda-863e-4fb1-a98c-9b325a8c0df7\") " pod="openshift-ingress-canary/ingress-canary-h9lck" Nov 02 22:52:20.067971 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.067883 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0237eeb1-8c85-475e-8580-6dec324de705-installation-pull-secrets\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:20.068034 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.068000 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/46eae5da-cf0b-4663-bdeb-f8220fa08c05-image-registry-private-configuration\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:20.068080 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.068031 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-default-certificate\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:20.068345 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.068322 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0237eeb1-8c85-475e-8580-6dec324de705-image-registry-private-configuration\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:20.068871 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.068849 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9c4b6fbd-74cf-41c6-bbae-92748ac25e1e-serving-cert\") pod \"console-operator-7cd56b8dbc-c2gxc\" (UID: \"9c4b6fbd-74cf-41c6-bbae-92748ac25e1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c2gxc" Nov 02 22:52:20.069178 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.069158 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/60832943-bf1f-4685-be00-08152c423b08-serving-cert\") pod \"service-ca-operator-544d749c8c-p66wt\" (UID: \"60832943-bf1f-4685-be00-08152c423b08\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p66wt" Nov 02 22:52:20.126379 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.124008 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncmnm\" (UniqueName: \"kubernetes.io/projected/4b9cd8c7-228b-451a-afe4-3af7a3bf0a42-kube-api-access-ncmnm\") pod \"kas-connection-checker-685895c7c9-mtrcx\" (UID: \"4b9cd8c7-228b-451a-afe4-3af7a3bf0a42\") " pod="kube-system/kas-connection-checker-685895c7c9-mtrcx" Nov 02 22:52:20.126379 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.124085 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkzvb\" (UniqueName: \"kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-kube-api-access-nkzvb\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:20.126379 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.124992 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws5j7\" (UniqueName: \"kubernetes.io/projected/b858466f-3dcd-419c-9495-4dbb89e8d51b-kube-api-access-ws5j7\") pod \"insights-operator-7c4986d449-xszmw\" (UID: \"b858466f-3dcd-419c-9495-4dbb89e8d51b\") " pod="openshift-insights/insights-operator-7c4986d449-xszmw" Nov 02 22:52:20.126379 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.125783 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqdk5\" (UniqueName: \"kubernetes.io/projected/0d008a27-d5ba-49cb-b3fb-abdecd723abd-kube-api-access-fqdk5\") pod \"volume-data-source-validator-58c9f5f46d-867x9\" (UID: \"0d008a27-d5ba-49cb-b3fb-abdecd723abd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-867x9" Nov 02 22:52:20.127586 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.127205 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89zw7\" (UniqueName: \"kubernetes.io/projected/9c4b6fbd-74cf-41c6-bbae-92748ac25e1e-kube-api-access-89zw7\") pod \"console-operator-7cd56b8dbc-c2gxc\" (UID: \"9c4b6fbd-74cf-41c6-bbae-92748ac25e1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c2gxc" Nov 02 22:52:20.128767 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.128416 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24smc\" (UniqueName: \"kubernetes.io/projected/fbcab4d0-9dd6-4682-be01-c4f589697e95-kube-api-access-24smc\") pod \"kas-connection-checker-685895c7c9-pxpxp\" (UID: \"fbcab4d0-9dd6-4682-be01-c4f589697e95\") " pod="kube-system/kas-connection-checker-685895c7c9-pxpxp" Nov 02 22:52:20.129390 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.129097 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxxqc\" (UniqueName: \"kubernetes.io/projected/236450f1-d9fa-488b-9022-ab4f5517549f-kube-api-access-vxxqc\") pod \"network-check-source-6f78d59f99-6zkx2\" (UID: \"236450f1-d9fa-488b-9022-ab4f5517549f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6zkx2" Nov 02 22:52:20.129918 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.129848 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjljw\" (UniqueName: \"kubernetes.io/projected/df418d6f-b9d3-4019-967c-9511be123e1f-kube-api-access-jjljw\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ht25s\" (UID: \"df418d6f-b9d3-4019-967c-9511be123e1f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ht25s" Nov 02 22:52:20.130023 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.129953 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-bound-sa-token\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:20.132707 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.132288 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-bound-sa-token\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:20.132707 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.132410 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stv25\" (UniqueName: \"kubernetes.io/projected/42d598be-a6f1-486e-9ac3-2f5ece575f89-kube-api-access-stv25\") pod \"kas-connection-checker-685895c7c9-fdj8t\" (UID: \"42d598be-a6f1-486e-9ac3-2f5ece575f89\") " pod="kube-system/kas-connection-checker-685895c7c9-fdj8t" Nov 02 22:52:20.132852 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.132754 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f4r4\" (UniqueName: \"kubernetes.io/projected/105406c8-6800-4562-9651-962bee9f8d73-kube-api-access-9f4r4\") pod \"dns-default-pvc5f\" (UID: \"105406c8-6800-4562-9651-962bee9f8d73\") " pod="openshift-dns/dns-default-pvc5f" Nov 02 22:52:20.133642 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.133616 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t8ms\" (UniqueName: \"kubernetes.io/projected/ac4a7fda-863e-4fb1-a98c-9b325a8c0df7-kube-api-access-9t8ms\") pod \"ingress-canary-h9lck\" (UID: \"ac4a7fda-863e-4fb1-a98c-9b325a8c0df7\") " pod="openshift-ingress-canary/ingress-canary-h9lck" Nov 02 22:52:20.133910 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.133867 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmk5v\" (UniqueName: \"kubernetes.io/projected/60832943-bf1f-4685-be00-08152c423b08-kube-api-access-pmk5v\") pod \"service-ca-operator-544d749c8c-p66wt\" (UID: \"60832943-bf1f-4685-be00-08152c423b08\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p66wt" Nov 02 22:52:20.134013 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.133993 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwnxj\" (UniqueName: \"kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-kube-api-access-nwnxj\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:20.134259 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.134238 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnt9q\" (UniqueName: \"kubernetes.io/projected/15789a0d-e030-45e0-a129-41de738ef767-kube-api-access-jnt9q\") pod \"cluster-monitoring-operator-884576c57-mbmf9\" (UID: \"15789a0d-e030-45e0-a129-41de738ef767\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbmf9" Nov 02 22:52:20.134356 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.134312 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md7ml\" (UniqueName: \"kubernetes.io/projected/75f7d1d7-88e4-41e0-bdd3-a43df363b408-kube-api-access-md7ml\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:20.134775 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.134752 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt847\" (UniqueName: \"kubernetes.io/projected/00b95c29-7df8-4602-bd0b-06b6281d5dc1-kube-api-access-zt847\") pod \"cluster-samples-operator-6ddb6c8b94-cgpkk\" (UID: \"00b95c29-7df8-4602-bd0b-06b6281d5dc1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cgpkk" Nov 02 22:52:20.163382 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.163359 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-fdj8t" Nov 02 22:52:20.168117 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.168097 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-mtrcx" Nov 02 22:52:20.174286 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.174265 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-pxpxp" Nov 02 22:52:20.208269 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.208234 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ht25s" Nov 02 22:52:20.220188 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.219998 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-867x9" Nov 02 22:52:20.232530 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.227672 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p66wt" Nov 02 22:52:20.234352 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.234187 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-c2gxc" Nov 02 22:52:20.268749 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.268521 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-xszmw" Nov 02 22:52:20.275490 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.275161 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6zkx2" Nov 02 22:52:20.284640 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.284434 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h9lck" Nov 02 22:52:20.548283 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.548248 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-mtrcx"] Nov 02 22:52:20.548615 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:52:20.548581 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b9cd8c7_228b_451a_afe4_3af7a3bf0a42.slice/crio-ee0f5d8810757aa96638f8b91b9e18c1185af1ff8cb70d801bfedb7b95ddda50 WatchSource:0}: Error finding container ee0f5d8810757aa96638f8b91b9e18c1185af1ff8cb70d801bfedb7b95ddda50: Status 404 returned error can't find the container with id ee0f5d8810757aa96638f8b91b9e18c1185af1ff8cb70d801bfedb7b95ddda50 Nov 02 22:52:20.549465 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.549443 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-pxpxp"] Nov 02 22:52:20.549558 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:52:20.549440 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d598be_a6f1_486e_9ac3_2f5ece575f89.slice/crio-47f55c937c827013ee62eecf35fb78175c77ee2729e44dbf3f8d37893e51eb7d WatchSource:0}: Error finding container 47f55c937c827013ee62eecf35fb78175c77ee2729e44dbf3f8d37893e51eb7d: Status 404 returned error can't find the container with id 47f55c937c827013ee62eecf35fb78175c77ee2729e44dbf3f8d37893e51eb7d Nov 02 22:52:20.550543 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.550490 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fdj8t"] Nov 02 22:52:20.552335 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:52:20.552247 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbcab4d0_9dd6_4682_be01_c4f589697e95.slice/crio-87ead83b47d5f3fc7380e32329e4f44fd3f61c3f216506076b57c9c75269f0d5 WatchSource:0}: Error finding container 87ead83b47d5f3fc7380e32329e4f44fd3f61c3f216506076b57c9c75269f0d5: Status 404 returned error can't find the container with id 87ead83b47d5f3fc7380e32329e4f44fd3f61c3f216506076b57c9c75269f0d5 Nov 02 22:52:20.567643 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.567610 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-registry-tls\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:20.567741 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.567672 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-registry-tls\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:20.567741 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.567703 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6c7074b-4caa-43fb-9fca-14ebb5abf958-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kkzvx\" (UID: \"b6c7074b-4caa-43fb-9fca-14ebb5abf958\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkzvx" Nov 02 22:52:20.567818 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.567768 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f7d1d7-88e4-41e0-bdd3-a43df363b408-service-ca-bundle\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:20.567871 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.567822 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/105406c8-6800-4562-9651-962bee9f8d73-metrics-tls\") pod \"dns-default-pvc5f\" (UID: \"105406c8-6800-4562-9651-962bee9f8d73\") " pod="openshift-dns/dns-default-pvc5f" Nov 02 22:52:20.567939 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.567865 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/15789a0d-e030-45e0-a129-41de738ef767-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mbmf9\" (UID: \"15789a0d-e030-45e0-a129-41de738ef767\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbmf9" Nov 02 22:52:20.567939 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.567915 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/00b95c29-7df8-4602-bd0b-06b6281d5dc1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-cgpkk\" (UID: \"00b95c29-7df8-4602-bd0b-06b6281d5dc1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cgpkk" Nov 02 22:52:20.568041 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.567962 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-metrics-certs\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:20.568115 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:20.568099 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 22:52:20.568172 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:20.568161 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-metrics-certs podName:75f7d1d7-88e4-41e0-bdd3-a43df363b408 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:21.568143562 +0000 UTC m=+33.242720360 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-metrics-certs") pod "router-default-896ccfb6c-q4rg5" (UID: "75f7d1d7-88e4-41e0-bdd3-a43df363b408") : secret "router-metrics-certs-default" not found Nov 02 22:52:20.568234 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:20.568227 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 22:52:20.568280 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:20.568239 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b5b8cd649-z4rbs: secret "image-registry-tls" not found Nov 02 22:52:20.568280 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:20.568271 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-registry-tls podName:46eae5da-cf0b-4663-bdeb-f8220fa08c05 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:21.56826124 +0000 UTC m=+33.242838023 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-registry-tls") pod "image-registry-7b5b8cd649-z4rbs" (UID: "46eae5da-cf0b-4663-bdeb-f8220fa08c05") : secret "image-registry-tls" not found Nov 02 22:52:20.568372 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:20.568327 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 22:52:20.568372 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:20.568337 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-777f69544d-d75n6: secret "image-registry-tls" not found Nov 02 22:52:20.568372 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:20.568365 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-registry-tls podName:0237eeb1-8c85-475e-8580-6dec324de705 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:21.568355932 +0000 UTC m=+33.242932713 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-registry-tls") pod "image-registry-777f69544d-d75n6" (UID: "0237eeb1-8c85-475e-8580-6dec324de705") : secret "image-registry-tls" not found Nov 02 22:52:20.568516 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:20.568421 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 22:52:20.568516 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:20.568477 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6c7074b-4caa-43fb-9fca-14ebb5abf958-networking-console-plugin-cert podName:b6c7074b-4caa-43fb-9fca-14ebb5abf958 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:21.568464977 +0000 UTC m=+33.243041755 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b6c7074b-4caa-43fb-9fca-14ebb5abf958-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kkzvx" (UID: "b6c7074b-4caa-43fb-9fca-14ebb5abf958") : secret "networking-console-plugin-cert" not found Nov 02 22:52:20.568609 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:20.568545 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75f7d1d7-88e4-41e0-bdd3-a43df363b408-service-ca-bundle podName:75f7d1d7-88e4-41e0-bdd3-a43df363b408 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:21.568535472 +0000 UTC m=+33.243112262 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/75f7d1d7-88e4-41e0-bdd3-a43df363b408-service-ca-bundle") pod "router-default-896ccfb6c-q4rg5" (UID: "75f7d1d7-88e4-41e0-bdd3-a43df363b408") : configmap references non-existent config key: service-ca.crt Nov 02 22:52:20.568667 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:20.568609 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 22:52:20.568667 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:20.568640 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/105406c8-6800-4562-9651-962bee9f8d73-metrics-tls podName:105406c8-6800-4562-9651-962bee9f8d73 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:21.568630245 +0000 UTC m=+33.243207040 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/105406c8-6800-4562-9651-962bee9f8d73-metrics-tls") pod "dns-default-pvc5f" (UID: "105406c8-6800-4562-9651-962bee9f8d73") : secret "dns-default-metrics-tls" not found Nov 02 22:52:20.568767 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:20.568697 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 22:52:20.568767 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:20.568730 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15789a0d-e030-45e0-a129-41de738ef767-cluster-monitoring-operator-tls podName:15789a0d-e030-45e0-a129-41de738ef767 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:21.568717719 +0000 UTC m=+33.243294498 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/15789a0d-e030-45e0-a129-41de738ef767-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-mbmf9" (UID: "15789a0d-e030-45e0-a129-41de738ef767") : secret "cluster-monitoring-operator-tls" not found Nov 02 22:52:20.568860 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:20.568782 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 22:52:20.568860 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:20.568811 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00b95c29-7df8-4602-bd0b-06b6281d5dc1-samples-operator-tls podName:00b95c29-7df8-4602-bd0b-06b6281d5dc1 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:21.568802054 +0000 UTC m=+33.243378833 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/00b95c29-7df8-4602-bd0b-06b6281d5dc1-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-cgpkk" (UID: "00b95c29-7df8-4602-bd0b-06b6281d5dc1") : secret "samples-operator-tls" not found Nov 02 22:52:20.592826 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.592794 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ht25s"] Nov 02 22:52:20.652319 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.652264 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-867x9"] Nov 02 22:52:20.655143 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.655055 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-c2gxc"] Nov 02 22:52:20.659980 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.658239 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-p66wt"] Nov 02 22:52:20.660311 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:52:20.660171 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60832943_bf1f_4685_be00_08152c423b08.slice/crio-9332cac3ad414a7a953cd8f44d53bed498e0df65d2ec5b79aae1ba771af997e6 WatchSource:0}: Error finding container 9332cac3ad414a7a953cd8f44d53bed498e0df65d2ec5b79aae1ba771af997e6: Status 404 returned error can't find the container with id 9332cac3ad414a7a953cd8f44d53bed498e0df65d2ec5b79aae1ba771af997e6 Nov 02 22:52:20.723572 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:52:20.723532 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac4a7fda_863e_4fb1_a98c_9b325a8c0df7.slice/crio-b1a2da6bba00ba8c0594bb3dbbeb3cca05136ff2fdbe870e87d5d1cbbfa6254a WatchSource:0}: Error finding container b1a2da6bba00ba8c0594bb3dbbeb3cca05136ff2fdbe870e87d5d1cbbfa6254a: Status 404 returned error can't find the container with id b1a2da6bba00ba8c0594bb3dbbeb3cca05136ff2fdbe870e87d5d1cbbfa6254a Nov 02 22:52:20.725053 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.725030 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h9lck"] Nov 02 22:52:20.725787 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:52:20.725764 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb858466f_3dcd_419c_9495_4dbb89e8d51b.slice/crio-f38c14908ece618cc8f271a7b83d2be46b29f52d2bcf1ca5db920bf6436246f8 WatchSource:0}: Error finding container f38c14908ece618cc8f271a7b83d2be46b29f52d2bcf1ca5db920bf6436246f8: Status 404 returned error can't find the container with id f38c14908ece618cc8f271a7b83d2be46b29f52d2bcf1ca5db920bf6436246f8 Nov 02 22:52:20.727950 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.727931 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-xszmw"] Nov 02 22:52:20.836516 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:52:20.836487 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod236450f1_d9fa_488b_9022_ab4f5517549f.slice/crio-055a42e181d047b2d0d0a3eb5266a6afecee615a52a63186bcdfc22ce49167c3 WatchSource:0}: Error finding container 055a42e181d047b2d0d0a3eb5266a6afecee615a52a63186bcdfc22ce49167c3: Status 404 returned error can't find the container with id 055a42e181d047b2d0d0a3eb5266a6afecee615a52a63186bcdfc22ce49167c3 Nov 02 22:52:20.838751 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.838731 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-6zkx2"] Nov 02 22:52:20.851193 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.851171 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:52:20.905562 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.905535 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 22:52:20.905984 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:20.905881 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-n2bfg\"" Nov 02 22:52:21.020931 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:21.020827 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h9lck" event={"ID":"ac4a7fda-863e-4fb1-a98c-9b325a8c0df7","Type":"ContainerStarted","Data":"b1a2da6bba00ba8c0594bb3dbbeb3cca05136ff2fdbe870e87d5d1cbbfa6254a"} Nov 02 22:52:21.021962 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:21.021939 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6zkx2" event={"ID":"236450f1-d9fa-488b-9022-ab4f5517549f","Type":"ContainerStarted","Data":"055a42e181d047b2d0d0a3eb5266a6afecee615a52a63186bcdfc22ce49167c3"} Nov 02 22:52:21.023004 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:21.022982 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-c2gxc" event={"ID":"9c4b6fbd-74cf-41c6-bbae-92748ac25e1e","Type":"ContainerStarted","Data":"318c3a34f01581248a349ccfe0a15df08396d1a44a40b4c33f781061059a387e"} Nov 02 22:52:21.024351 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:21.024312 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-fdj8t" event={"ID":"42d598be-a6f1-486e-9ac3-2f5ece575f89","Type":"ContainerStarted","Data":"348ac8fe53e029a6ff5d4c314c830245276676a7c4f07db81ef9f98e3264f89d"} Nov 02 22:52:21.024454 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:21.024365 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-fdj8t" event={"ID":"42d598be-a6f1-486e-9ac3-2f5ece575f89","Type":"ContainerStarted","Data":"47f55c937c827013ee62eecf35fb78175c77ee2729e44dbf3f8d37893e51eb7d"} Nov 02 22:52:21.025731 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:21.025705 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-mtrcx" event={"ID":"4b9cd8c7-228b-451a-afe4-3af7a3bf0a42","Type":"ContainerStarted","Data":"9c8b10056d5f9969890ba4f834e42f48fd358301d66ef6c15618434c55853188"} Nov 02 22:52:21.025820 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:21.025741 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-mtrcx" event={"ID":"4b9cd8c7-228b-451a-afe4-3af7a3bf0a42","Type":"ContainerStarted","Data":"ee0f5d8810757aa96638f8b91b9e18c1185af1ff8cb70d801bfedb7b95ddda50"} Nov 02 22:52:21.026752 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:21.026731 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p66wt" event={"ID":"60832943-bf1f-4685-be00-08152c423b08","Type":"ContainerStarted","Data":"9332cac3ad414a7a953cd8f44d53bed498e0df65d2ec5b79aae1ba771af997e6"} Nov 02 22:52:21.027764 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:21.027749 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-867x9" event={"ID":"0d008a27-d5ba-49cb-b3fb-abdecd723abd","Type":"ContainerStarted","Data":"de924197b3afd07e8c6b2d9ab15d26217b20b69c974fcc935146dd25598040e8"} Nov 02 22:52:21.028747 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:21.028723 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-xszmw" event={"ID":"b858466f-3dcd-419c-9495-4dbb89e8d51b","Type":"ContainerStarted","Data":"f38c14908ece618cc8f271a7b83d2be46b29f52d2bcf1ca5db920bf6436246f8"} Nov 02 22:52:21.029689 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:21.029669 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ht25s" event={"ID":"df418d6f-b9d3-4019-967c-9511be123e1f","Type":"ContainerStarted","Data":"696172ee14d01e9d35bdac289d3290e2adc725e4cafa322f60dc04e3cb208c47"} Nov 02 22:52:21.030850 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:21.030832 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-pxpxp" event={"ID":"fbcab4d0-9dd6-4682-be01-c4f589697e95","Type":"ContainerStarted","Data":"322f48a0b6e9980ac561a48fc79b9458d22a0289ee6426464ee4ddec9807b99b"} Nov 02 22:52:21.030942 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:21.030860 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-pxpxp" event={"ID":"fbcab4d0-9dd6-4682-be01-c4f589697e95","Type":"ContainerStarted","Data":"87ead83b47d5f3fc7380e32329e4f44fd3f61c3f216506076b57c9c75269f0d5"} Nov 02 22:52:21.578067 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:21.578027 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-metrics-certs\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:21.578216 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:21.578082 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-registry-tls\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:21.578216 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:21.578115 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-registry-tls\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:21.578216 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:21.578142 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6c7074b-4caa-43fb-9fca-14ebb5abf958-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kkzvx\" (UID: \"b6c7074b-4caa-43fb-9fca-14ebb5abf958\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkzvx" Nov 02 22:52:21.578216 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:21.578175 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 22:52:21.578216 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:21.578190 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f7d1d7-88e4-41e0-bdd3-a43df363b408-service-ca-bundle\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:21.578446 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:21.578218 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 22:52:21.578446 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:21.578237 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b5b8cd649-z4rbs: secret "image-registry-tls" not found Nov 02 22:52:21.578446 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:21.578239 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-metrics-certs podName:75f7d1d7-88e4-41e0-bdd3-a43df363b408 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:23.578221842 +0000 UTC m=+35.252798634 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-metrics-certs") pod "router-default-896ccfb6c-q4rg5" (UID: "75f7d1d7-88e4-41e0-bdd3-a43df363b408") : secret "router-metrics-certs-default" not found Nov 02 22:52:21.578446 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:21.578277 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 22:52:21.578446 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:21.578296 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-777f69544d-d75n6: secret "image-registry-tls" not found Nov 02 22:52:21.578446 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:21.578304 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 22:52:21.578446 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:21.578305 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75f7d1d7-88e4-41e0-bdd3-a43df363b408-service-ca-bundle podName:75f7d1d7-88e4-41e0-bdd3-a43df363b408 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:23.578289792 +0000 UTC m=+35.252866577 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/75f7d1d7-88e4-41e0-bdd3-a43df363b408-service-ca-bundle") pod "router-default-896ccfb6c-q4rg5" (UID: "75f7d1d7-88e4-41e0-bdd3-a43df363b408") : configmap references non-existent config key: service-ca.crt Nov 02 22:52:21.578446 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:21.578363 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-registry-tls podName:46eae5da-cf0b-4663-bdeb-f8220fa08c05 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:23.578352843 +0000 UTC m=+35.252929625 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-registry-tls") pod "image-registry-7b5b8cd649-z4rbs" (UID: "46eae5da-cf0b-4663-bdeb-f8220fa08c05") : secret "image-registry-tls" not found Nov 02 22:52:21.578446 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:21.578386 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-registry-tls podName:0237eeb1-8c85-475e-8580-6dec324de705 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:23.578377341 +0000 UTC m=+35.252954133 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-registry-tls") pod "image-registry-777f69544d-d75n6" (UID: "0237eeb1-8c85-475e-8580-6dec324de705") : secret "image-registry-tls" not found Nov 02 22:52:21.578446 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:21.578400 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6c7074b-4caa-43fb-9fca-14ebb5abf958-networking-console-plugin-cert podName:b6c7074b-4caa-43fb-9fca-14ebb5abf958 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:23.578392511 +0000 UTC m=+35.252969295 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b6c7074b-4caa-43fb-9fca-14ebb5abf958-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kkzvx" (UID: "b6c7074b-4caa-43fb-9fca-14ebb5abf958") : secret "networking-console-plugin-cert" not found Nov 02 22:52:21.578446 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:21.578414 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/105406c8-6800-4562-9651-962bee9f8d73-metrics-tls\") pod \"dns-default-pvc5f\" (UID: \"105406c8-6800-4562-9651-962bee9f8d73\") " pod="openshift-dns/dns-default-pvc5f" Nov 02 22:52:21.578848 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:21.578455 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/15789a0d-e030-45e0-a129-41de738ef767-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mbmf9\" (UID: \"15789a0d-e030-45e0-a129-41de738ef767\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbmf9" Nov 02 22:52:21.578848 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:21.578491 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/00b95c29-7df8-4602-bd0b-06b6281d5dc1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-cgpkk\" (UID: \"00b95c29-7df8-4602-bd0b-06b6281d5dc1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cgpkk" Nov 02 22:52:21.578848 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:21.578536 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 22:52:21.578848 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:21.578572 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 22:52:21.578848 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:21.578588 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 22:52:21.578848 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:21.578577 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/105406c8-6800-4562-9651-962bee9f8d73-metrics-tls podName:105406c8-6800-4562-9651-962bee9f8d73 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:23.578564471 +0000 UTC m=+35.253141267 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/105406c8-6800-4562-9651-962bee9f8d73-metrics-tls") pod "dns-default-pvc5f" (UID: "105406c8-6800-4562-9651-962bee9f8d73") : secret "dns-default-metrics-tls" not found Nov 02 22:52:21.578848 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:21.578633 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15789a0d-e030-45e0-a129-41de738ef767-cluster-monitoring-operator-tls podName:15789a0d-e030-45e0-a129-41de738ef767 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:23.578618193 +0000 UTC m=+35.253194990 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/15789a0d-e030-45e0-a129-41de738ef767-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-mbmf9" (UID: "15789a0d-e030-45e0-a129-41de738ef767") : secret "cluster-monitoring-operator-tls" not found Nov 02 22:52:21.578848 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:21.578650 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00b95c29-7df8-4602-bd0b-06b6281d5dc1-samples-operator-tls podName:00b95c29-7df8-4602-bd0b-06b6281d5dc1 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:23.578640732 +0000 UTC m=+35.253217557 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/00b95c29-7df8-4602-bd0b-06b6281d5dc1-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-cgpkk" (UID: "00b95c29-7df8-4602-bd0b-06b6281d5dc1") : secret "samples-operator-tls" not found Nov 02 22:52:21.780717 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:21.780678 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs\") pod \"network-metrics-daemon-9zqrz\" (UID: \"fc69f07f-4001-4951-aff8-d3d6671cc5d4\") " pod="openshift-multus/network-metrics-daemon-9zqrz" Nov 02 22:52:21.780870 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:21.780722 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd96552a-1a78-4fac-8faf-71d36f010905-original-pull-secret\") pod \"global-pull-secret-syncer-k4gfs\" (UID: \"fd96552a-1a78-4fac-8faf-71d36f010905\") " pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:52:21.780870 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:21.780851 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 22:52:21.781044 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:21.781019 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs podName:fc69f07f-4001-4951-aff8-d3d6671cc5d4 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:53.780996997 +0000 UTC m=+65.455573778 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs") pod "network-metrics-daemon-9zqrz" (UID: "fc69f07f-4001-4951-aff8-d3d6671cc5d4") : secret "metrics-daemon-secret" not found Nov 02 22:52:21.783169 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:21.783150 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd96552a-1a78-4fac-8faf-71d36f010905-original-pull-secret\") pod \"global-pull-secret-syncer-k4gfs\" (UID: \"fd96552a-1a78-4fac-8faf-71d36f010905\") " pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:52:21.882285 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:21.881724 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7nrp\" (UniqueName: \"kubernetes.io/projected/e3a3bc57-6813-441c-bdc0-d39eb8214c61-kube-api-access-q7nrp\") pod \"network-check-target-rmgpt\" (UID: \"e3a3bc57-6813-441c-bdc0-d39eb8214c61\") " pod="openshift-network-diagnostics/network-check-target-rmgpt" Nov 02 22:52:21.885294 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:21.885224 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7nrp\" (UniqueName: \"kubernetes.io/projected/e3a3bc57-6813-441c-bdc0-d39eb8214c61-kube-api-access-q7nrp\") pod \"network-check-target-rmgpt\" (UID: \"e3a3bc57-6813-441c-bdc0-d39eb8214c61\") " pod="openshift-network-diagnostics/network-check-target-rmgpt" Nov 02 22:52:21.986169 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:21.986140 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rmgpt" Nov 02 22:52:22.074249 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:22.074097 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:52:22.109176 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:22.109129 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-fdj8t" podStartSLOduration=441.109114401 podStartE2EDuration="7m21.109114401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:45:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:21.992851183 +0000 UTC m=+33.667427985" watchObservedRunningTime="2026-11-02 22:52:22.109114401 +0000 UTC m=+33.783691183" Nov 02 22:52:22.110765 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:22.110685 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-mtrcx" podStartSLOduration=441.110671922 podStartE2EDuration="7m21.110671922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:45:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:22.109063431 +0000 UTC m=+33.783640234" watchObservedRunningTime="2026-11-02 22:52:22.110671922 +0000 UTC m=+33.785248723" Nov 02 22:52:22.237115 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:22.237046 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-pxpxp" podStartSLOduration=441.23702053 podStartE2EDuration="7m21.23702053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:45:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:22.232440912 +0000 UTC m=+33.907017712" watchObservedRunningTime="2026-11-02 22:52:22.23702053 +0000 UTC m=+33.911597331" Nov 02 22:52:22.383616 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:22.383500 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rmgpt"] Nov 02 22:52:22.387987 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:52:22.387955 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3a3bc57_6813_441c_bdc0_d39eb8214c61.slice/crio-54ad588bb8e86ed38f7a08e74739b501f3663a3a0f5cda683afea2c00d975db2 WatchSource:0}: Error finding container 54ad588bb8e86ed38f7a08e74739b501f3663a3a0f5cda683afea2c00d975db2: Status 404 returned error can't find the container with id 54ad588bb8e86ed38f7a08e74739b501f3663a3a0f5cda683afea2c00d975db2 Nov 02 22:52:22.537797 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:52:22.537758 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd96552a_1a78_4fac_8faf_71d36f010905.slice/crio-bae187d60c71e81ac2558a9049659dfeb1817e080bfab2d74e4c2e79695056ee WatchSource:0}: Error finding container bae187d60c71e81ac2558a9049659dfeb1817e080bfab2d74e4c2e79695056ee: Status 404 returned error can't find the container with id bae187d60c71e81ac2558a9049659dfeb1817e080bfab2d74e4c2e79695056ee Nov 02 22:52:22.539099 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:22.538158 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-k4gfs"] Nov 02 22:52:23.041794 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:23.041019 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-k4gfs" event={"ID":"fd96552a-1a78-4fac-8faf-71d36f010905","Type":"ContainerStarted","Data":"bae187d60c71e81ac2558a9049659dfeb1817e080bfab2d74e4c2e79695056ee"} Nov 02 22:52:23.045704 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:23.045644 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rmgpt" event={"ID":"e3a3bc57-6813-441c-bdc0-d39eb8214c61","Type":"ContainerStarted","Data":"54ad588bb8e86ed38f7a08e74739b501f3663a3a0f5cda683afea2c00d975db2"} Nov 02 22:52:23.600802 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:23.600718 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-registry-tls\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:23.600802 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:23.600785 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-registry-tls\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:23.601172 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:23.600817 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6c7074b-4caa-43fb-9fca-14ebb5abf958-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kkzvx\" (UID: \"b6c7074b-4caa-43fb-9fca-14ebb5abf958\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkzvx" Nov 02 22:52:23.601172 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:23.600878 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f7d1d7-88e4-41e0-bdd3-a43df363b408-service-ca-bundle\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:23.601172 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:23.600941 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/105406c8-6800-4562-9651-962bee9f8d73-metrics-tls\") pod \"dns-default-pvc5f\" (UID: \"105406c8-6800-4562-9651-962bee9f8d73\") " pod="openshift-dns/dns-default-pvc5f" Nov 02 22:52:23.601172 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:23.600981 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/15789a0d-e030-45e0-a129-41de738ef767-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mbmf9\" (UID: \"15789a0d-e030-45e0-a129-41de738ef767\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbmf9" Nov 02 22:52:23.601172 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:23.601018 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/00b95c29-7df8-4602-bd0b-06b6281d5dc1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-cgpkk\" (UID: \"00b95c29-7df8-4602-bd0b-06b6281d5dc1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cgpkk" Nov 02 22:52:23.601172 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:23.601064 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-metrics-certs\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:23.601545 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:23.601192 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 22:52:23.601545 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:23.601251 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-metrics-certs podName:75f7d1d7-88e4-41e0-bdd3-a43df363b408 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:27.601238153 +0000 UTC m=+39.275814932 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-metrics-certs") pod "router-default-896ccfb6c-q4rg5" (UID: "75f7d1d7-88e4-41e0-bdd3-a43df363b408") : secret "router-metrics-certs-default" not found Nov 02 22:52:23.601662 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:23.601634 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 22:52:23.601662 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:23.601648 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b5b8cd649-z4rbs: secret "image-registry-tls" not found Nov 02 22:52:23.601772 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:23.601684 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-registry-tls podName:46eae5da-cf0b-4663-bdeb-f8220fa08c05 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:27.601672833 +0000 UTC m=+39.276249611 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-registry-tls") pod "image-registry-7b5b8cd649-z4rbs" (UID: "46eae5da-cf0b-4663-bdeb-f8220fa08c05") : secret "image-registry-tls" not found Nov 02 22:52:23.601772 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:23.601724 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 22:52:23.601772 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:23.601730 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-777f69544d-d75n6: secret "image-registry-tls" not found Nov 02 22:52:23.601772 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:23.601748 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-registry-tls podName:0237eeb1-8c85-475e-8580-6dec324de705 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:27.60174247 +0000 UTC m=+39.276319247 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-registry-tls") pod "image-registry-777f69544d-d75n6" (UID: "0237eeb1-8c85-475e-8580-6dec324de705") : secret "image-registry-tls" not found Nov 02 22:52:23.602008 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:23.601786 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 22:52:23.602008 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:23.601824 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6c7074b-4caa-43fb-9fca-14ebb5abf958-networking-console-plugin-cert podName:b6c7074b-4caa-43fb-9fca-14ebb5abf958 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:27.601817978 +0000 UTC m=+39.276394757 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b6c7074b-4caa-43fb-9fca-14ebb5abf958-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kkzvx" (UID: "b6c7074b-4caa-43fb-9fca-14ebb5abf958") : secret "networking-console-plugin-cert" not found Nov 02 22:52:23.602008 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:23.601864 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75f7d1d7-88e4-41e0-bdd3-a43df363b408-service-ca-bundle podName:75f7d1d7-88e4-41e0-bdd3-a43df363b408 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:27.601858606 +0000 UTC m=+39.276435384 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/75f7d1d7-88e4-41e0-bdd3-a43df363b408-service-ca-bundle") pod "router-default-896ccfb6c-q4rg5" (UID: "75f7d1d7-88e4-41e0-bdd3-a43df363b408") : configmap references non-existent config key: service-ca.crt Nov 02 22:52:23.602008 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:23.601920 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 22:52:23.602008 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:23.601948 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/105406c8-6800-4562-9651-962bee9f8d73-metrics-tls podName:105406c8-6800-4562-9651-962bee9f8d73 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:27.601938155 +0000 UTC m=+39.276514938 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/105406c8-6800-4562-9651-962bee9f8d73-metrics-tls") pod "dns-default-pvc5f" (UID: "105406c8-6800-4562-9651-962bee9f8d73") : secret "dns-default-metrics-tls" not found Nov 02 22:52:23.602008 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:23.601995 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 22:52:23.602288 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:23.602024 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15789a0d-e030-45e0-a129-41de738ef767-cluster-monitoring-operator-tls podName:15789a0d-e030-45e0-a129-41de738ef767 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:27.602015272 +0000 UTC m=+39.276592052 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/15789a0d-e030-45e0-a129-41de738ef767-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-mbmf9" (UID: "15789a0d-e030-45e0-a129-41de738ef767") : secret "cluster-monitoring-operator-tls" not found Nov 02 22:52:23.602288 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:23.602075 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 22:52:23.602288 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:23.602102 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00b95c29-7df8-4602-bd0b-06b6281d5dc1-samples-operator-tls podName:00b95c29-7df8-4602-bd0b-06b6281d5dc1 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:27.602094041 +0000 UTC m=+39.276670823 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/00b95c29-7df8-4602-bd0b-06b6281d5dc1-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-cgpkk" (UID: "00b95c29-7df8-4602-bd0b-06b6281d5dc1") : secret "samples-operator-tls" not found Nov 02 22:52:24.057428 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:24.057346 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-867x9" event={"ID":"0d008a27-d5ba-49cb-b3fb-abdecd723abd","Type":"ContainerStarted","Data":"8fe1fd425eca8c40e7949d87b2cc00e88733476f0961c0cbb49fcf69570eff77"} Nov 02 22:52:26.072812 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:26.072783 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/0.log" Nov 02 22:52:26.073245 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:26.072829 2575 generic.go:356] "Generic (PLEG): container finished" podID="9c4b6fbd-74cf-41c6-bbae-92748ac25e1e" containerID="e0316854b97d3d79be97cdc4709958579959670124f28426f177fc7fc251d6c4" exitCode=255 Nov 02 22:52:26.073245 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:26.072925 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-c2gxc" event={"ID":"9c4b6fbd-74cf-41c6-bbae-92748ac25e1e","Type":"ContainerDied","Data":"e0316854b97d3d79be97cdc4709958579959670124f28426f177fc7fc251d6c4"} Nov 02 22:52:26.073352 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:26.073297 2575 scope.go:122] "RemoveContainer" containerID="e0316854b97d3d79be97cdc4709958579959670124f28426f177fc7fc251d6c4" Nov 02 22:52:26.190613 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:26.190561 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-867x9" podStartSLOduration=416.811824822 podStartE2EDuration="6m59.190544522s" totalImagesPullingTime="2.3787197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:45:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:24.175107015 +0000 UTC m=+35.849683818" watchObservedRunningTime="2026-11-02 22:52:26.190544522 +0000 UTC m=+37.865121319" Nov 02 22:52:27.083914 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:27.083091 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p66wt" event={"ID":"60832943-bf1f-4685-be00-08152c423b08","Type":"ContainerStarted","Data":"207bec755b31eb4ef158daaf761bd0e8fe353dd48459ead26c5aa9b9ab0fe67a"} Nov 02 22:52:27.088095 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:27.088058 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-xszmw" event={"ID":"b858466f-3dcd-419c-9495-4dbb89e8d51b","Type":"ContainerStarted","Data":"8495045bd8628a15e3796c689e1715ce7627d22e65221455858e5862a91aed6c"} Nov 02 22:52:27.091981 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:27.091951 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ht25s" event={"ID":"df418d6f-b9d3-4019-967c-9511be123e1f","Type":"ContainerStarted","Data":"fcd8f266982c92e7a3d3a16cef84c06c4ad5a3efbf549823e26e26e1c034495c"} Nov 02 22:52:27.098663 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:27.098589 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/1.log" Nov 02 22:52:27.099164 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:27.099073 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/0.log" Nov 02 22:52:27.099164 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:27.099111 2575 generic.go:356] "Generic (PLEG): container finished" podID="9c4b6fbd-74cf-41c6-bbae-92748ac25e1e" containerID="61bc91e4752e543b447d5adcc7109202c5002dfd3c1c92a91e17d4830f0014e5" exitCode=255 Nov 02 22:52:27.099311 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:27.099177 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-c2gxc" event={"ID":"9c4b6fbd-74cf-41c6-bbae-92748ac25e1e","Type":"ContainerDied","Data":"61bc91e4752e543b447d5adcc7109202c5002dfd3c1c92a91e17d4830f0014e5"} Nov 02 22:52:27.099311 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:27.099223 2575 scope.go:122] "RemoveContainer" containerID="e0316854b97d3d79be97cdc4709958579959670124f28426f177fc7fc251d6c4" Nov 02 22:52:27.099806 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:27.099605 2575 scope.go:122] "RemoveContainer" containerID="61bc91e4752e543b447d5adcc7109202c5002dfd3c1c92a91e17d4830f0014e5" Nov 02 22:52:27.099806 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:27.099797 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-c2gxc_openshift-console-operator(9c4b6fbd-74cf-41c6-bbae-92748ac25e1e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-c2gxc" podUID="9c4b6fbd-74cf-41c6-bbae-92748ac25e1e" Nov 02 22:52:27.318925 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:27.317824 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p66wt" podStartSLOduration=414.54013298 podStartE2EDuration="7m0.317801889s" totalImagesPullingTime="5.777668909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:45:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:27.199248944 +0000 UTC m=+38.873825745" watchObservedRunningTime="2026-11-02 22:52:27.317801889 +0000 UTC m=+38.992378691" Nov 02 22:52:27.318925 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:27.318192 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ht25s" podStartSLOduration=414.901919432 podStartE2EDuration="7m0.318182171s" totalImagesPullingTime="5.416262739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:45:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:27.317096491 +0000 UTC m=+38.991673291" watchObservedRunningTime="2026-11-02 22:52:27.318182171 +0000 UTC m=+38.992758973" Nov 02 22:52:27.645782 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:27.645747 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-registry-tls\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:27.645976 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:27.645793 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6c7074b-4caa-43fb-9fca-14ebb5abf958-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kkzvx\" (UID: \"b6c7074b-4caa-43fb-9fca-14ebb5abf958\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkzvx" Nov 02 22:52:27.645976 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:27.645928 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 22:52:27.645976 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:27.645948 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 22:52:27.645976 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:27.645972 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-777f69544d-d75n6: secret "image-registry-tls" not found Nov 02 22:52:27.646179 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:27.645971 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f7d1d7-88e4-41e0-bdd3-a43df363b408-service-ca-bundle\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:27.646179 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:27.645990 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6c7074b-4caa-43fb-9fca-14ebb5abf958-networking-console-plugin-cert podName:b6c7074b-4caa-43fb-9fca-14ebb5abf958 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:35.645972071 +0000 UTC m=+47.320548850 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b6c7074b-4caa-43fb-9fca-14ebb5abf958-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kkzvx" (UID: "b6c7074b-4caa-43fb-9fca-14ebb5abf958") : secret "networking-console-plugin-cert" not found Nov 02 22:52:27.646179 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:27.646028 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-registry-tls podName:0237eeb1-8c85-475e-8580-6dec324de705 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:35.646009096 +0000 UTC m=+47.320585877 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-registry-tls") pod "image-registry-777f69544d-d75n6" (UID: "0237eeb1-8c85-475e-8580-6dec324de705") : secret "image-registry-tls" not found Nov 02 22:52:27.646179 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:27.646069 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75f7d1d7-88e4-41e0-bdd3-a43df363b408-service-ca-bundle podName:75f7d1d7-88e4-41e0-bdd3-a43df363b408 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:35.64605581 +0000 UTC m=+47.320632606 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/75f7d1d7-88e4-41e0-bdd3-a43df363b408-service-ca-bundle") pod "router-default-896ccfb6c-q4rg5" (UID: "75f7d1d7-88e4-41e0-bdd3-a43df363b408") : configmap references non-existent config key: service-ca.crt Nov 02 22:52:27.646179 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:27.646094 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/105406c8-6800-4562-9651-962bee9f8d73-metrics-tls\") pod \"dns-default-pvc5f\" (UID: \"105406c8-6800-4562-9651-962bee9f8d73\") " pod="openshift-dns/dns-default-pvc5f" Nov 02 22:52:27.646179 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:27.646141 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/15789a0d-e030-45e0-a129-41de738ef767-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mbmf9\" (UID: \"15789a0d-e030-45e0-a129-41de738ef767\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbmf9" Nov 02 22:52:27.646179 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:27.646174 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/00b95c29-7df8-4602-bd0b-06b6281d5dc1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-cgpkk\" (UID: \"00b95c29-7df8-4602-bd0b-06b6281d5dc1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cgpkk" Nov 02 22:52:27.646520 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:27.646199 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 22:52:27.646520 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:27.646227 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-metrics-certs\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:27.646520 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:27.646242 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/105406c8-6800-4562-9651-962bee9f8d73-metrics-tls podName:105406c8-6800-4562-9651-962bee9f8d73 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:35.646227184 +0000 UTC m=+47.320803967 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/105406c8-6800-4562-9651-962bee9f8d73-metrics-tls") pod "dns-default-pvc5f" (UID: "105406c8-6800-4562-9651-962bee9f8d73") : secret "dns-default-metrics-tls" not found Nov 02 22:52:27.646520 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:27.646284 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-registry-tls\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:27.646520 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:27.646298 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 22:52:27.646520 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:27.646338 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-metrics-certs podName:75f7d1d7-88e4-41e0-bdd3-a43df363b408 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:35.646325638 +0000 UTC m=+47.320902435 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-metrics-certs") pod "router-default-896ccfb6c-q4rg5" (UID: "75f7d1d7-88e4-41e0-bdd3-a43df363b408") : secret "router-metrics-certs-default" not found Nov 02 22:52:27.646520 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:27.646390 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 22:52:27.646520 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:27.646400 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 22:52:27.646520 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:27.646411 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b5b8cd649-z4rbs: secret "image-registry-tls" not found Nov 02 22:52:27.646520 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:27.646437 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15789a0d-e030-45e0-a129-41de738ef767-cluster-monitoring-operator-tls podName:15789a0d-e030-45e0-a129-41de738ef767 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:35.646424167 +0000 UTC m=+47.321000960 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/15789a0d-e030-45e0-a129-41de738ef767-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-mbmf9" (UID: "15789a0d-e030-45e0-a129-41de738ef767") : secret "cluster-monitoring-operator-tls" not found Nov 02 22:52:27.646520 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:27.646458 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-registry-tls podName:46eae5da-cf0b-4663-bdeb-f8220fa08c05 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:35.646445736 +0000 UTC m=+47.321022521 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-registry-tls") pod "image-registry-7b5b8cd649-z4rbs" (UID: "46eae5da-cf0b-4663-bdeb-f8220fa08c05") : secret "image-registry-tls" not found Nov 02 22:52:27.646520 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:27.646392 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 22:52:27.646520 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:27.646492 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00b95c29-7df8-4602-bd0b-06b6281d5dc1-samples-operator-tls podName:00b95c29-7df8-4602-bd0b-06b6281d5dc1 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:35.64648314 +0000 UTC m=+47.321059921 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/00b95c29-7df8-4602-bd0b-06b6281d5dc1-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-cgpkk" (UID: "00b95c29-7df8-4602-bd0b-06b6281d5dc1") : secret "samples-operator-tls" not found Nov 02 22:52:27.782713 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:27.782648 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-xszmw" podStartSLOduration=416.502006096 podStartE2EDuration="7m1.782627451s" totalImagesPullingTime="5.280621355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:45:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:27.560073171 +0000 UTC m=+39.234649981" watchObservedRunningTime="2026-11-02 22:52:27.782627451 +0000 UTC m=+39.457204252" Nov 02 22:52:27.783107 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:27.783086 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-sv7kh"] Nov 02 22:52:27.786326 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:27.786287 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sv7kh" Nov 02 22:52:27.842195 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:27.842149 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vbflc\"" Nov 02 22:52:27.842195 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:27.842159 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 22:52:27.842548 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:27.842527 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sv7kh"] Nov 02 22:52:27.932492 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:27.932403 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 22:52:27.948650 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:27.948607 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9f98166f-8051-4dbd-9856-db692932050b-crio-socket\") pod \"insights-runtime-extractor-sv7kh\" (UID: \"9f98166f-8051-4dbd-9856-db692932050b\") " pod="openshift-insights/insights-runtime-extractor-sv7kh" Nov 02 22:52:27.948810 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:27.948746 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f98166f-8051-4dbd-9856-db692932050b-data-volume\") pod \"insights-runtime-extractor-sv7kh\" (UID: \"9f98166f-8051-4dbd-9856-db692932050b\") " pod="openshift-insights/insights-runtime-extractor-sv7kh" Nov 02 22:52:27.948810 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:27.948795 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f98166f-8051-4dbd-9856-db692932050b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sv7kh\" (UID: \"9f98166f-8051-4dbd-9856-db692932050b\") " pod="openshift-insights/insights-runtime-extractor-sv7kh" Nov 02 22:52:27.948994 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:27.948975 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85k6p\" (UniqueName: \"kubernetes.io/projected/9f98166f-8051-4dbd-9856-db692932050b-kube-api-access-85k6p\") pod \"insights-runtime-extractor-sv7kh\" (UID: \"9f98166f-8051-4dbd-9856-db692932050b\") " pod="openshift-insights/insights-runtime-extractor-sv7kh" Nov 02 22:52:27.949063 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:27.949014 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f98166f-8051-4dbd-9856-db692932050b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sv7kh\" (UID: \"9f98166f-8051-4dbd-9856-db692932050b\") " pod="openshift-insights/insights-runtime-extractor-sv7kh" Nov 02 22:52:27.979331 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:27.979284 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-f62lt"] Nov 02 22:52:27.981847 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:27.981821 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-f62lt" Nov 02 22:52:27.989119 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:27.989085 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8mxfd"] Nov 02 22:52:27.992535 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:27.992513 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8mxfd" Nov 02 22:52:28.038838 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.038716 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 22:52:28.038838 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.038788 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 22:52:28.040529 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.040476 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-f62lt"] Nov 02 22:52:28.044331 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.044008 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8mxfd"] Nov 02 22:52:28.044331 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.044295 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-45bns\"" Nov 02 22:52:28.050474 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.050445 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j4hf\" (UniqueName: \"kubernetes.io/projected/50d13ae8-4ab5-4ca7-a11a-ad632da19669-kube-api-access-5j4hf\") pod \"migrator-66df5d496b-f62lt\" (UID: \"50d13ae8-4ab5-4ca7-a11a-ad632da19669\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-f62lt" Nov 02 22:52:28.050588 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.050495 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9f98166f-8051-4dbd-9856-db692932050b-crio-socket\") pod \"insights-runtime-extractor-sv7kh\" (UID: \"9f98166f-8051-4dbd-9856-db692932050b\") " pod="openshift-insights/insights-runtime-extractor-sv7kh" Nov 02 22:52:28.050674 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.050648 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f98166f-8051-4dbd-9856-db692932050b-data-volume\") pod \"insights-runtime-extractor-sv7kh\" (UID: \"9f98166f-8051-4dbd-9856-db692932050b\") " pod="openshift-insights/insights-runtime-extractor-sv7kh" Nov 02 22:52:28.050726 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.050697 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f98166f-8051-4dbd-9856-db692932050b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sv7kh\" (UID: \"9f98166f-8051-4dbd-9856-db692932050b\") " pod="openshift-insights/insights-runtime-extractor-sv7kh" Nov 02 22:52:28.050848 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.050827 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85k6p\" (UniqueName: \"kubernetes.io/projected/9f98166f-8051-4dbd-9856-db692932050b-kube-api-access-85k6p\") pod \"insights-runtime-extractor-sv7kh\" (UID: \"9f98166f-8051-4dbd-9856-db692932050b\") " pod="openshift-insights/insights-runtime-extractor-sv7kh" Nov 02 22:52:28.050980 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.050868 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f98166f-8051-4dbd-9856-db692932050b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sv7kh\" (UID: \"9f98166f-8051-4dbd-9856-db692932050b\") " pod="openshift-insights/insights-runtime-extractor-sv7kh" Nov 02 22:52:28.051056 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:28.051036 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 22:52:28.051107 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:28.051098 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f98166f-8051-4dbd-9856-db692932050b-insights-runtime-extractor-tls podName:9f98166f-8051-4dbd-9856-db692932050b nodeName:}" failed. No retries permitted until 2026-11-02 22:52:28.551080689 +0000 UTC m=+40.225657480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f98166f-8051-4dbd-9856-db692932050b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sv7kh" (UID: "9f98166f-8051-4dbd-9856-db692932050b") : secret "insights-runtime-extractor-tls" not found Nov 02 22:52:28.051181 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.051162 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9f98166f-8051-4dbd-9856-db692932050b-crio-socket\") pod \"insights-runtime-extractor-sv7kh\" (UID: \"9f98166f-8051-4dbd-9856-db692932050b\") " pod="openshift-insights/insights-runtime-extractor-sv7kh" Nov 02 22:52:28.051502 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.051482 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f98166f-8051-4dbd-9856-db692932050b-data-volume\") pod \"insights-runtime-extractor-sv7kh\" (UID: \"9f98166f-8051-4dbd-9856-db692932050b\") " pod="openshift-insights/insights-runtime-extractor-sv7kh" Nov 02 22:52:28.051573 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:28.051560 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 22:52:28.051627 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:28.051606 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f98166f-8051-4dbd-9856-db692932050b-kube-rbac-proxy-cm podName:9f98166f-8051-4dbd-9856-db692932050b nodeName:}" failed. No retries permitted until 2026-11-02 22:52:28.551589808 +0000 UTC m=+40.226166601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9f98166f-8051-4dbd-9856-db692932050b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sv7kh" (UID: "9f98166f-8051-4dbd-9856-db692932050b") : configmap "kube-rbac-proxy" not found Nov 02 22:52:28.064922 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.064470 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-lx8nq-mmb4p"] Nov 02 22:52:28.067160 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.067139 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lx8nq-mmb4p" Nov 02 22:52:28.105903 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.105862 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h9lck" event={"ID":"ac4a7fda-863e-4fb1-a98c-9b325a8c0df7","Type":"ContainerStarted","Data":"1edf2b6d1daf7cdf6403326df828f62bf2f8315cb9f68ae0f3495f09a9757494"} Nov 02 22:52:28.108174 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.108137 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/1.log" Nov 02 22:52:28.108930 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.108639 2575 scope.go:122] "RemoveContainer" containerID="61bc91e4752e543b447d5adcc7109202c5002dfd3c1c92a91e17d4830f0014e5" Nov 02 22:52:28.108930 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:28.108843 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-c2gxc_openshift-console-operator(9c4b6fbd-74cf-41c6-bbae-92748ac25e1e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-c2gxc" podUID="9c4b6fbd-74cf-41c6-bbae-92748ac25e1e" Nov 02 22:52:28.119031 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.119001 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85k6p\" (UniqueName: \"kubernetes.io/projected/9f98166f-8051-4dbd-9856-db692932050b-kube-api-access-85k6p\") pod \"insights-runtime-extractor-sv7kh\" (UID: \"9f98166f-8051-4dbd-9856-db692932050b\") " pod="openshift-insights/insights-runtime-extractor-sv7kh" Nov 02 22:52:28.122167 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.122140 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-lx8nq-mmb4p"] Nov 02 22:52:28.124877 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.124840 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-nxb2c\"" Nov 02 22:52:28.151726 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.151695 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d12c211b-f607-43b9-b621-12f061f109d9-service-ca-bundle\") pod \"periodic-gathering-lx8nq-mmb4p\" (UID: \"d12c211b-f607-43b9-b621-12f061f109d9\") " pod="openshift-insights/periodic-gathering-lx8nq-mmb4p" Nov 02 22:52:28.152094 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.151754 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6flqc\" (UniqueName: \"kubernetes.io/projected/0b6b45c3-5275-4877-a84b-15e602100c40-kube-api-access-6flqc\") pod \"migrator-66df5d496b-8mxfd\" (UID: \"0b6b45c3-5275-4877-a84b-15e602100c40\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8mxfd" Nov 02 22:52:28.152094 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.151834 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j4hf\" (UniqueName: \"kubernetes.io/projected/50d13ae8-4ab5-4ca7-a11a-ad632da19669-kube-api-access-5j4hf\") pod \"migrator-66df5d496b-f62lt\" (UID: \"50d13ae8-4ab5-4ca7-a11a-ad632da19669\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-f62lt" Nov 02 22:52:28.152094 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.151871 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx9mw\" (UniqueName: \"kubernetes.io/projected/d12c211b-f607-43b9-b621-12f061f109d9-kube-api-access-bx9mw\") pod \"periodic-gathering-lx8nq-mmb4p\" (UID: \"d12c211b-f607-43b9-b621-12f061f109d9\") " pod="openshift-insights/periodic-gathering-lx8nq-mmb4p" Nov 02 22:52:28.152094 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.151950 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d12c211b-f607-43b9-b621-12f061f109d9-trusted-ca-bundle\") pod \"periodic-gathering-lx8nq-mmb4p\" (UID: \"d12c211b-f607-43b9-b621-12f061f109d9\") " pod="openshift-insights/periodic-gathering-lx8nq-mmb4p" Nov 02 22:52:28.152094 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.152019 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d12c211b-f607-43b9-b621-12f061f109d9-archives-path\") pod \"periodic-gathering-lx8nq-mmb4p\" (UID: \"d12c211b-f607-43b9-b621-12f061f109d9\") " pod="openshift-insights/periodic-gathering-lx8nq-mmb4p" Nov 02 22:52:28.218261 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.218172 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j4hf\" (UniqueName: \"kubernetes.io/projected/50d13ae8-4ab5-4ca7-a11a-ad632da19669-kube-api-access-5j4hf\") pod \"migrator-66df5d496b-f62lt\" (UID: \"50d13ae8-4ab5-4ca7-a11a-ad632da19669\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-f62lt" Nov 02 22:52:28.254959 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.252711 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d12c211b-f607-43b9-b621-12f061f109d9-service-ca-bundle\") pod \"periodic-gathering-lx8nq-mmb4p\" (UID: \"d12c211b-f607-43b9-b621-12f061f109d9\") " pod="openshift-insights/periodic-gathering-lx8nq-mmb4p" Nov 02 22:52:28.254959 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.252776 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6flqc\" (UniqueName: \"kubernetes.io/projected/0b6b45c3-5275-4877-a84b-15e602100c40-kube-api-access-6flqc\") pod \"migrator-66df5d496b-8mxfd\" (UID: \"0b6b45c3-5275-4877-a84b-15e602100c40\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8mxfd" Nov 02 22:52:28.254959 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.252924 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx9mw\" (UniqueName: \"kubernetes.io/projected/d12c211b-f607-43b9-b621-12f061f109d9-kube-api-access-bx9mw\") pod \"periodic-gathering-lx8nq-mmb4p\" (UID: \"d12c211b-f607-43b9-b621-12f061f109d9\") " pod="openshift-insights/periodic-gathering-lx8nq-mmb4p" Nov 02 22:52:28.254959 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.253046 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d12c211b-f607-43b9-b621-12f061f109d9-trusted-ca-bundle\") pod \"periodic-gathering-lx8nq-mmb4p\" (UID: \"d12c211b-f607-43b9-b621-12f061f109d9\") " pod="openshift-insights/periodic-gathering-lx8nq-mmb4p" Nov 02 22:52:28.254959 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.253173 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d12c211b-f607-43b9-b621-12f061f109d9-archives-path\") pod \"periodic-gathering-lx8nq-mmb4p\" (UID: \"d12c211b-f607-43b9-b621-12f061f109d9\") " pod="openshift-insights/periodic-gathering-lx8nq-mmb4p" Nov 02 22:52:28.254959 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.254132 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d12c211b-f607-43b9-b621-12f061f109d9-archives-path\") pod \"periodic-gathering-lx8nq-mmb4p\" (UID: \"d12c211b-f607-43b9-b621-12f061f109d9\") " pod="openshift-insights/periodic-gathering-lx8nq-mmb4p" Nov 02 22:52:28.254959 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.254414 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d12c211b-f607-43b9-b621-12f061f109d9-trusted-ca-bundle\") pod \"periodic-gathering-lx8nq-mmb4p\" (UID: \"d12c211b-f607-43b9-b621-12f061f109d9\") " pod="openshift-insights/periodic-gathering-lx8nq-mmb4p" Nov 02 22:52:28.255311 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.255231 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d12c211b-f607-43b9-b621-12f061f109d9-service-ca-bundle\") pod \"periodic-gathering-lx8nq-mmb4p\" (UID: \"d12c211b-f607-43b9-b621-12f061f109d9\") " pod="openshift-insights/periodic-gathering-lx8nq-mmb4p" Nov 02 22:52:28.295395 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.295214 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-f62lt" Nov 02 22:52:28.315395 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.313407 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx9mw\" (UniqueName: \"kubernetes.io/projected/d12c211b-f607-43b9-b621-12f061f109d9-kube-api-access-bx9mw\") pod \"periodic-gathering-lx8nq-mmb4p\" (UID: \"d12c211b-f607-43b9-b621-12f061f109d9\") " pod="openshift-insights/periodic-gathering-lx8nq-mmb4p" Nov 02 22:52:28.315395 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.314065 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6flqc\" (UniqueName: \"kubernetes.io/projected/0b6b45c3-5275-4877-a84b-15e602100c40-kube-api-access-6flqc\") pod \"migrator-66df5d496b-8mxfd\" (UID: \"0b6b45c3-5275-4877-a84b-15e602100c40\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8mxfd" Nov 02 22:52:28.392457 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.392428 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lx8nq-mmb4p" Nov 02 22:52:28.460450 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.456958 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-h9lck" podStartSLOduration=3.005690999 podStartE2EDuration="9.45694s" totalImagesPullingTime="6.451249001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:52:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:28.455153835 +0000 UTC m=+40.129730636" watchObservedRunningTime="2026-11-02 22:52:28.45694 +0000 UTC m=+40.131516803" Nov 02 22:52:28.557420 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.557315 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f98166f-8051-4dbd-9856-db692932050b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sv7kh\" (UID: \"9f98166f-8051-4dbd-9856-db692932050b\") " pod="openshift-insights/insights-runtime-extractor-sv7kh" Nov 02 22:52:28.557584 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.557464 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f98166f-8051-4dbd-9856-db692932050b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sv7kh\" (UID: \"9f98166f-8051-4dbd-9856-db692932050b\") " pod="openshift-insights/insights-runtime-extractor-sv7kh" Nov 02 22:52:28.557584 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:28.557509 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 22:52:28.557584 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:28.557579 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f98166f-8051-4dbd-9856-db692932050b-insights-runtime-extractor-tls podName:9f98166f-8051-4dbd-9856-db692932050b nodeName:}" failed. No retries permitted until 2026-11-02 22:52:29.557561353 +0000 UTC m=+41.232138131 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f98166f-8051-4dbd-9856-db692932050b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sv7kh" (UID: "9f98166f-8051-4dbd-9856-db692932050b") : secret "insights-runtime-extractor-tls" not found Nov 02 22:52:28.557735 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:28.557595 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 22:52:28.557735 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:28.557646 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f98166f-8051-4dbd-9856-db692932050b-kube-rbac-proxy-cm podName:9f98166f-8051-4dbd-9856-db692932050b nodeName:}" failed. No retries permitted until 2026-11-02 22:52:29.557631171 +0000 UTC m=+41.232207954 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9f98166f-8051-4dbd-9856-db692932050b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sv7kh" (UID: "9f98166f-8051-4dbd-9856-db692932050b") : configmap "kube-rbac-proxy" not found Nov 02 22:52:28.604979 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.604939 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8mxfd" Nov 02 22:52:28.690977 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:52:28.690451 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50d13ae8_4ab5_4ca7_a11a_ad632da19669.slice/crio-924af956a0c2c62f133a420c317fb5dd6c2d5bbfde22749bfceb02dc429d5b67 WatchSource:0}: Error finding container 924af956a0c2c62f133a420c317fb5dd6c2d5bbfde22749bfceb02dc429d5b67: Status 404 returned error can't find the container with id 924af956a0c2c62f133a420c317fb5dd6c2d5bbfde22749bfceb02dc429d5b67 Nov 02 22:52:28.691703 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.691662 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-f62lt"] Nov 02 22:52:28.769249 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:52:28.769217 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd12c211b_f607_43b9_b621_12f061f109d9.slice/crio-923bdca363bdecec9681204d8c7d61a686ad35aad15b349b72375be1ca7ac71e WatchSource:0}: Error finding container 923bdca363bdecec9681204d8c7d61a686ad35aad15b349b72375be1ca7ac71e: Status 404 returned error can't find the container with id 923bdca363bdecec9681204d8c7d61a686ad35aad15b349b72375be1ca7ac71e Nov 02 22:52:28.771210 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.771187 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-lx8nq-mmb4p"] Nov 02 22:52:28.979585 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:52:28.979552 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b6b45c3_5275_4877_a84b_15e602100c40.slice/crio-387a216b6d58d9b749b3b5ca2cc899e1729c8ed618fd1fd4517f1cc35170c68e WatchSource:0}: Error finding container 387a216b6d58d9b749b3b5ca2cc899e1729c8ed618fd1fd4517f1cc35170c68e: Status 404 returned error can't find the container with id 387a216b6d58d9b749b3b5ca2cc899e1729c8ed618fd1fd4517f1cc35170c68e Nov 02 22:52:28.982641 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:28.982613 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8mxfd"] Nov 02 22:52:29.113862 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:29.113777 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6zkx2" event={"ID":"236450f1-d9fa-488b-9022-ab4f5517549f","Type":"ContainerStarted","Data":"f58a1501eec6db2628be83901dcc3c79dd1c3aa7a62176ab62d7a9acccbe7556"} Nov 02 22:52:29.116777 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:29.116717 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rmgpt" event={"ID":"e3a3bc57-6813-441c-bdc0-d39eb8214c61","Type":"ContainerStarted","Data":"5b5283f366cac2326a004f2447c00083a4846299be93c03c4875179878d410dc"} Nov 02 22:52:29.116932 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:29.116789 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rmgpt" Nov 02 22:52:29.117945 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:29.117919 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8mxfd" event={"ID":"0b6b45c3-5275-4877-a84b-15e602100c40","Type":"ContainerStarted","Data":"387a216b6d58d9b749b3b5ca2cc899e1729c8ed618fd1fd4517f1cc35170c68e"} Nov 02 22:52:29.119572 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:29.119543 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lx8nq-mmb4p" event={"ID":"d12c211b-f607-43b9-b621-12f061f109d9","Type":"ContainerStarted","Data":"29d97269a85c034b372c9aa93b7ff5f2295abd6535fe741e8ab8c382e87f6393"} Nov 02 22:52:29.119677 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:29.119587 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lx8nq-mmb4p" event={"ID":"d12c211b-f607-43b9-b621-12f061f109d9","Type":"ContainerStarted","Data":"923bdca363bdecec9681204d8c7d61a686ad35aad15b349b72375be1ca7ac71e"} Nov 02 22:52:29.120826 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:29.120804 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-f62lt" event={"ID":"50d13ae8-4ab5-4ca7-a11a-ad632da19669","Type":"ContainerStarted","Data":"924af956a0c2c62f133a420c317fb5dd6c2d5bbfde22749bfceb02dc429d5b67"} Nov 02 22:52:29.231419 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:29.231360 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6zkx2" podStartSLOduration=413.839367812 podStartE2EDuration="7m1.231346957s" totalImagesPullingTime="7.391979145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:45:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:29.230465957 +0000 UTC m=+40.905042761" watchObservedRunningTime="2026-11-02 22:52:29.231346957 +0000 UTC m=+40.905923756" Nov 02 22:52:29.350548 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:29.350484 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rmgpt" podStartSLOduration=33.818321375 podStartE2EDuration="40.350469019s" totalImagesPullingTime="6.532147644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:51:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:29.348952879 +0000 UTC m=+41.023529680" watchObservedRunningTime="2026-11-02 22:52:29.350469019 +0000 UTC m=+41.025045818" Nov 02 22:52:29.364933 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:29.364837 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-hrdqf"] Nov 02 22:52:29.368070 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:29.367488 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-hrdqf" Nov 02 22:52:29.420692 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:29.420600 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-hrdqf"] Nov 02 22:52:29.422138 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:29.422108 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 02 22:52:29.422138 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:29.422124 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 02 22:52:29.422138 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:29.422137 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 02 22:52:29.422424 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:29.422218 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 02 22:52:29.422424 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:29.422276 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-qg26z\"" Nov 02 22:52:29.469914 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:29.468311 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-lx8nq-mmb4p" podStartSLOduration=1.468291763 podStartE2EDuration="1.468291763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:52:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:29.468096676 +0000 UTC m=+41.142673477" watchObservedRunningTime="2026-11-02 22:52:29.468291763 +0000 UTC m=+41.142868565" Nov 02 22:52:29.469914 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:29.468366 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/79e2a74e-667b-4851-9e8a-759cb2a91f98-signing-key\") pod \"service-ca-5fffc97bdf-hrdqf\" (UID: \"79e2a74e-667b-4851-9e8a-759cb2a91f98\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hrdqf" Nov 02 22:52:29.469914 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:29.468431 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx46q\" (UniqueName: \"kubernetes.io/projected/79e2a74e-667b-4851-9e8a-759cb2a91f98-kube-api-access-lx46q\") pod \"service-ca-5fffc97bdf-hrdqf\" (UID: \"79e2a74e-667b-4851-9e8a-759cb2a91f98\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hrdqf" Nov 02 22:52:29.469914 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:29.468589 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/79e2a74e-667b-4851-9e8a-759cb2a91f98-signing-cabundle\") pod \"service-ca-5fffc97bdf-hrdqf\" (UID: \"79e2a74e-667b-4851-9e8a-759cb2a91f98\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hrdqf" Nov 02 22:52:29.569275 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:29.569240 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/79e2a74e-667b-4851-9e8a-759cb2a91f98-signing-key\") pod \"service-ca-5fffc97bdf-hrdqf\" (UID: \"79e2a74e-667b-4851-9e8a-759cb2a91f98\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hrdqf" Nov 02 22:52:29.569439 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:29.569294 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f98166f-8051-4dbd-9856-db692932050b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sv7kh\" (UID: \"9f98166f-8051-4dbd-9856-db692932050b\") " pod="openshift-insights/insights-runtime-extractor-sv7kh" Nov 02 22:52:29.569439 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:29.569325 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx46q\" (UniqueName: \"kubernetes.io/projected/79e2a74e-667b-4851-9e8a-759cb2a91f98-kube-api-access-lx46q\") pod \"service-ca-5fffc97bdf-hrdqf\" (UID: \"79e2a74e-667b-4851-9e8a-759cb2a91f98\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hrdqf" Nov 02 22:52:29.569439 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:29.569398 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/79e2a74e-667b-4851-9e8a-759cb2a91f98-signing-cabundle\") pod \"service-ca-5fffc97bdf-hrdqf\" (UID: \"79e2a74e-667b-4851-9e8a-759cb2a91f98\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hrdqf" Nov 02 22:52:29.569596 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:29.569546 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 22:52:29.569644 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:29.569597 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f98166f-8051-4dbd-9856-db692932050b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sv7kh\" (UID: \"9f98166f-8051-4dbd-9856-db692932050b\") " pod="openshift-insights/insights-runtime-extractor-sv7kh" Nov 02 22:52:29.569644 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:29.569634 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f98166f-8051-4dbd-9856-db692932050b-insights-runtime-extractor-tls podName:9f98166f-8051-4dbd-9856-db692932050b nodeName:}" failed. No retries permitted until 2026-11-02 22:52:31.569611284 +0000 UTC m=+43.244188081 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f98166f-8051-4dbd-9856-db692932050b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sv7kh" (UID: "9f98166f-8051-4dbd-9856-db692932050b") : secret "insights-runtime-extractor-tls" not found Nov 02 22:52:29.569745 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:29.569656 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 22:52:29.569745 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:29.569716 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f98166f-8051-4dbd-9856-db692932050b-kube-rbac-proxy-cm podName:9f98166f-8051-4dbd-9856-db692932050b nodeName:}" failed. No retries permitted until 2026-11-02 22:52:31.569700109 +0000 UTC m=+43.244276902 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9f98166f-8051-4dbd-9856-db692932050b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sv7kh" (UID: "9f98166f-8051-4dbd-9856-db692932050b") : configmap "kube-rbac-proxy" not found Nov 02 22:52:29.570167 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:29.570146 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/79e2a74e-667b-4851-9e8a-759cb2a91f98-signing-cabundle\") pod \"service-ca-5fffc97bdf-hrdqf\" (UID: \"79e2a74e-667b-4851-9e8a-759cb2a91f98\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hrdqf" Nov 02 22:52:29.572011 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:29.571991 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/79e2a74e-667b-4851-9e8a-759cb2a91f98-signing-key\") pod \"service-ca-5fffc97bdf-hrdqf\" (UID: \"79e2a74e-667b-4851-9e8a-759cb2a91f98\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hrdqf" Nov 02 22:52:29.628704 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:29.628641 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx46q\" (UniqueName: \"kubernetes.io/projected/79e2a74e-667b-4851-9e8a-759cb2a91f98-kube-api-access-lx46q\") pod \"service-ca-5fffc97bdf-hrdqf\" (UID: \"79e2a74e-667b-4851-9e8a-759cb2a91f98\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hrdqf" Nov 02 22:52:29.680428 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:29.680390 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-hrdqf" Nov 02 22:52:30.059615 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:30.059536 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-hrdqf"] Nov 02 22:52:30.067200 ip-10-0-130-211 kubenswrapper[2575]: W1102 22:52:30.067158 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e2a74e_667b_4851_9e8a_759cb2a91f98.slice/crio-31ea80300d42577ad470fb6f4391ef10c997d0836355f54625ad14cb535f4483 WatchSource:0}: Error finding container 31ea80300d42577ad470fb6f4391ef10c997d0836355f54625ad14cb535f4483: Status 404 returned error can't find the container with id 31ea80300d42577ad470fb6f4391ef10c997d0836355f54625ad14cb535f4483 Nov 02 22:52:30.129384 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:30.129350 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-hrdqf" event={"ID":"79e2a74e-667b-4851-9e8a-759cb2a91f98","Type":"ContainerStarted","Data":"31ea80300d42577ad470fb6f4391ef10c997d0836355f54625ad14cb535f4483"} Nov 02 22:52:30.235762 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:30.235106 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-c2gxc" Nov 02 22:52:30.235762 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:30.235160 2575 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-c2gxc" Nov 02 22:52:30.235762 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:30.235506 2575 scope.go:122] "RemoveContainer" containerID="61bc91e4752e543b447d5adcc7109202c5002dfd3c1c92a91e17d4830f0014e5" Nov 02 22:52:30.235762 ip-10-0-130-211 kubenswrapper[2575]: E1102 22:52:30.235694 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-c2gxc_openshift-console-operator(9c4b6fbd-74cf-41c6-bbae-92748ac25e1e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-c2gxc" podUID="9c4b6fbd-74cf-41c6-bbae-92748ac25e1e" Nov 02 22:52:30.962489 ip-10-0-130-211 kubenswrapper[2575]: I1102 22:52:30.962384 2575 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 22:52:30.962553 ip-10-0-130-211 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 22:52:30.973270 ip-10-0-130-211 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 22:52:30.973449 ip-10-0-130-211 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 22:52:30.973502 ip-10-0-130-211 systemd[1]: kubelet.service: Consumed 3.203s CPU time. Nov 02 22:52:30.989401 ip-10-0-130-211 systemd[1]: Starting Kubernetes Kubelet... Nov 02 22:52:31.062827 ip-10-0-130-211 kubenswrapper[5076]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:52:31.062827 ip-10-0-130-211 kubenswrapper[5076]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 22:52:31.062827 ip-10-0-130-211 kubenswrapper[5076]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:52:31.062827 ip-10-0-130-211 kubenswrapper[5076]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:52:31.064972 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.064872 5076 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:52:31.064972 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.064972 5076 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:52:31.065036 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.064977 5076 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:52:31.065036 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.064980 5076 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:52:31.065036 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.064983 5076 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:52:31.065036 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.064986 5076 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:52:31.065036 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.064988 5076 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:52:31.065036 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.064991 5076 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:52:31.065036 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.064994 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:52:31.065036 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.064996 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:52:31.065036 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.064999 5076 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:52:31.065036 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065011 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:52:31.065036 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065015 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:52:31.065036 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065017 5076 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:52:31.065036 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065025 5076 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:52:31.065036 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065028 5076 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:52:31.065036 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065031 5076 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:52:31.065036 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065034 5076 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:52:31.065036 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065037 5076 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:52:31.065036 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065040 5076 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:52:31.065036 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065043 5076 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:52:31.065513 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065046 5076 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:52:31.065513 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065050 5076 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:52:31.065513 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065056 5076 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:52:31.065513 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065059 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:52:31.065513 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065062 5076 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:52:31.065513 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065064 5076 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:52:31.065513 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065067 5076 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:52:31.065513 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065069 5076 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:52:31.065513 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065072 5076 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:52:31.065513 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065075 5076 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:52:31.065513 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065078 5076 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:52:31.065513 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065080 5076 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:52:31.065513 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065083 5076 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:52:31.065513 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065085 5076 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:52:31.065513 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065088 5076 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:52:31.065513 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065090 5076 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:52:31.065513 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065093 5076 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:52:31.065513 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065096 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:52:31.065513 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065098 5076 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:52:31.066003 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065100 5076 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:52:31.066003 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065103 5076 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:52:31.066003 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065105 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:52:31.066003 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065108 5076 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:52:31.066003 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065111 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:52:31.066003 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065115 5076 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:52:31.066003 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065118 5076 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:52:31.066003 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065120 5076 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:52:31.066003 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065123 5076 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:52:31.066003 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065126 5076 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:52:31.066003 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065128 5076 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:52:31.066003 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065131 5076 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:52:31.066003 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065133 5076 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:52:31.066003 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065135 5076 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:52:31.066003 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065138 5076 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:52:31.066003 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065141 5076 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:52:31.066003 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065144 5076 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:52:31.066003 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065146 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:52:31.066003 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065149 5076 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:52:31.066003 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065152 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:52:31.066483 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065154 5076 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:52:31.066483 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065157 5076 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:52:31.066483 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065159 5076 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:52:31.066483 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065162 5076 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:52:31.066483 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065164 5076 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:52:31.066483 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065167 5076 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:52:31.066483 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065170 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:52:31.066483 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065172 5076 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:52:31.066483 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065175 5076 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:52:31.066483 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065178 5076 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:52:31.066483 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065181 5076 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:52:31.066483 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065184 5076 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:52:31.066483 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065186 5076 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:52:31.066483 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065189 5076 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:52:31.066483 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065191 5076 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:52:31.066483 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065194 5076 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:52:31.066483 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065197 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:52:31.066483 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065200 5076 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:52:31.066483 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065202 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:52:31.066483 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065206 5076 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:52:31.067235 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065210 5076 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:52:31.067235 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065213 5076 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:52:31.067235 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065215 5076 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:52:31.067235 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065218 5076 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:52:31.067235 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065220 5076 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:52:31.067235 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065223 5076 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:52:31.067235 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065226 5076 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:52:31.067235 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065229 5076 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:52:31.067235 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065231 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:52:31.067235 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065234 5076 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:52:31.067235 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065237 5076 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:52:31.067235 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065239 5076 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:52:31.067235 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065242 5076 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:52:31.067235 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065244 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:52:31.067235 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065247 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:52:31.067235 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065249 5076 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:52:31.067235 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065252 5076 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:52:31.067235 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065255 5076 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:52:31.067235 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065257 5076 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:52:31.067960 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065260 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:52:31.067960 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065263 5076 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:52:31.067960 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065266 5076 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:52:31.067960 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065270 5076 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:52:31.067960 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065274 5076 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:52:31.067960 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065278 5076 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:52:31.067960 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.065282 5076 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:52:31.067960 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066088 5076 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:52:31.067960 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066096 5076 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:52:31.067960 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066100 5076 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:52:31.067960 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066102 5076 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:52:31.067960 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066105 5076 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:52:31.067960 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066109 5076 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:52:31.067960 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066112 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:52:31.067960 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066115 5076 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:52:31.067960 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066117 5076 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:52:31.067960 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066120 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:52:31.067960 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066123 5076 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:52:31.067960 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066125 5076 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:52:31.067960 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066128 5076 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:52:31.068465 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066131 5076 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:52:31.068465 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066133 5076 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:52:31.068465 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066136 5076 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:52:31.068465 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066138 5076 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:52:31.068465 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066141 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:52:31.068465 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066144 5076 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:52:31.068465 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066146 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:52:31.068465 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066149 5076 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:52:31.068465 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066152 5076 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:52:31.068465 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066155 5076 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:52:31.068465 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066157 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:52:31.068465 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066160 5076 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:52:31.068465 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066163 5076 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:52:31.068465 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066165 5076 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:52:31.068465 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066169 5076 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:52:31.068465 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066171 5076 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:52:31.068465 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066174 5076 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:52:31.068465 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066177 5076 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:52:31.068465 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066179 5076 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:52:31.068998 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066182 5076 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:52:31.068998 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066185 5076 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:52:31.068998 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066188 5076 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:52:31.068998 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066193 5076 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:52:31.068998 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066196 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:52:31.068998 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066199 5076 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:52:31.068998 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066204 5076 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:52:31.068998 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066208 5076 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:52:31.068998 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066211 5076 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:52:31.068998 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066215 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:52:31.068998 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066218 5076 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:52:31.068998 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066221 5076 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:52:31.068998 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066223 5076 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:52:31.068998 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066230 5076 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:52:31.068998 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066233 5076 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:52:31.068998 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066235 5076 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:52:31.068998 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066238 5076 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:52:31.068998 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066240 5076 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:52:31.069419 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066243 5076 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:52:31.069419 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066246 5076 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:52:31.069419 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066249 5076 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:52:31.069419 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066252 5076 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:52:31.069419 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066255 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:52:31.069419 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066257 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:52:31.069419 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066260 5076 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:52:31.069419 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066263 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:52:31.069419 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066265 5076 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:52:31.069419 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066269 5076 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:52:31.069419 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066271 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:52:31.069419 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066274 5076 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:52:31.069419 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066277 5076 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:52:31.069419 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066279 5076 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:52:31.069419 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066282 5076 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:52:31.069419 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066285 5076 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:52:31.069419 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066287 5076 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:52:31.069419 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066290 5076 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:52:31.069419 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066292 5076 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:52:31.069419 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066295 5076 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:52:31.069912 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066297 5076 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:52:31.069912 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066301 5076 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:52:31.069912 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066304 5076 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:52:31.069912 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066307 5076 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:52:31.069912 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066309 5076 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:52:31.069912 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066312 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:52:31.069912 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066315 5076 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:52:31.069912 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066317 5076 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:52:31.069912 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066320 5076 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:52:31.069912 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066322 5076 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:52:31.069912 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066325 5076 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:52:31.069912 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066328 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:52:31.069912 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066331 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:52:31.069912 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066333 5076 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:52:31.069912 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066336 5076 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:52:31.069912 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066339 5076 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:52:31.069912 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066341 5076 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:52:31.069912 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066343 5076 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:52:31.069912 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066346 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:52:31.069912 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066349 5076 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:52:31.070435 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066352 5076 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:52:31.070435 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066355 5076 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:52:31.070435 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066358 5076 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:52:31.070435 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066361 5076 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:52:31.070435 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066364 5076 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:52:31.070435 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066366 5076 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:52:31.070435 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066369 5076 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:52:31.070435 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066371 5076 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:52:31.070435 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066374 5076 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:52:31.070435 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066377 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:52:31.070435 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066380 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:52:31.070435 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066382 5076 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:52:31.070435 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066385 5076 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:52:31.070435 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066387 5076 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:52:31.070435 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066390 5076 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:52:31.070435 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.066394 5076 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:52:31.070435 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068130 5076 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:52:31.070435 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068140 5076 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:52:31.070435 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068145 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:52:31.070435 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068148 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:52:31.070978 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068151 5076 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:52:31.070978 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068155 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:52:31.070978 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068158 5076 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:52:31.070978 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068161 5076 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:52:31.070978 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068163 5076 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:52:31.070978 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068166 5076 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:52:31.070978 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068169 5076 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:52:31.070978 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068171 5076 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:52:31.070978 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068174 5076 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:52:31.070978 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068177 5076 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:52:31.070978 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068179 5076 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:52:31.070978 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068182 5076 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:52:31.070978 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068185 5076 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:52:31.070978 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068187 5076 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:52:31.070978 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068190 5076 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:52:31.070978 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068193 5076 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:52:31.070978 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068197 5076 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:52:31.070978 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068201 5076 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:52:31.070978 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068205 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:52:31.070978 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068208 5076 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:52:31.071478 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068211 5076 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:52:31.071478 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068214 5076 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:52:31.071478 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068217 5076 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:52:31.071478 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068219 5076 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:52:31.071478 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068222 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:52:31.071478 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068224 5076 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:52:31.071478 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068233 5076 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:52:31.071478 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068236 5076 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:52:31.071478 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068238 5076 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:52:31.071478 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068242 5076 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:52:31.071478 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068245 5076 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:52:31.071478 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068247 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:52:31.071478 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068250 5076 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:52:31.071478 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068253 5076 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:52:31.071478 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068256 5076 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:52:31.071478 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068258 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:52:31.071478 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068261 5076 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:52:31.071478 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068263 5076 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:52:31.071478 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068266 5076 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:52:31.071478 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068269 5076 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:52:31.072434 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068271 5076 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:52:31.072434 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068273 5076 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:52:31.072434 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068276 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:52:31.072434 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068279 5076 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:52:31.072434 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068281 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:52:31.072434 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068286 5076 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:52:31.072434 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068290 5076 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:52:31.072434 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068293 5076 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:52:31.072434 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068296 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:52:31.072434 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068299 5076 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:52:31.072434 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068302 5076 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:52:31.072434 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068305 5076 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:52:31.072434 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068308 5076 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:52:31.072434 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068310 5076 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:52:31.072434 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068313 5076 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:52:31.072434 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068315 5076 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:52:31.072434 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068318 5076 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:52:31.072434 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068322 5076 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:52:31.072434 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068325 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:52:31.072926 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068328 5076 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:52:31.072926 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068330 5076 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:52:31.072926 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068333 5076 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:52:31.072926 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068336 5076 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:52:31.072926 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068338 5076 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:52:31.072926 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068341 5076 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:52:31.072926 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068343 5076 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:52:31.072926 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068346 5076 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:52:31.072926 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068349 5076 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:52:31.072926 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068352 5076 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:52:31.072926 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068354 5076 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:52:31.072926 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068357 5076 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:52:31.072926 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068360 5076 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:52:31.072926 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068363 5076 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:52:31.072926 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068365 5076 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:52:31.072926 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068368 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:52:31.072926 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068370 5076 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:52:31.072926 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068373 5076 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:52:31.072926 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068375 5076 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:52:31.072926 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068378 5076 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:52:31.073411 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068381 5076 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:52:31.073411 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068384 5076 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:52:31.073411 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068386 5076 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:52:31.073411 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068389 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:52:31.073411 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068391 5076 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:52:31.073411 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068394 5076 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:52:31.073411 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068396 5076 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:52:31.073411 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068399 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:52:31.073411 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068402 5076 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:52:31.073411 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068404 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:52:31.073411 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068408 5076 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:52:31.073411 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068410 5076 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:52:31.073411 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068413 5076 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:52:31.073411 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068415 5076 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:52:31.073411 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068419 5076 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:52:31.073411 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068421 5076 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:52:31.073411 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068424 5076 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:52:31.073411 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068426 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:52:31.073411 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068429 5076 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:52:31.073851 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068432 5076 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:52:31.073851 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068434 5076 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:52:31.073851 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068437 5076 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:52:31.073851 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.068439 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:52:31.073851 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068533 5076 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 22:52:31.073851 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068540 5076 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 22:52:31.073851 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068546 5076 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 22:52:31.073851 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068551 5076 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 22:52:31.073851 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068555 5076 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 22:52:31.073851 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068559 5076 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 22:52:31.073851 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068563 5076 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 22:52:31.073851 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068567 5076 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 22:52:31.073851 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068570 5076 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 22:52:31.073851 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068574 5076 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 22:52:31.073851 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068578 5076 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 22:52:31.073851 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068581 5076 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 22:52:31.073851 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068584 5076 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 22:52:31.073851 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068587 5076 flags.go:64] FLAG: --cgroup-root="" Nov 02 22:52:31.073851 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068590 5076 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 22:52:31.073851 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068594 5076 flags.go:64] FLAG: --client-ca-file="" Nov 02 22:52:31.073851 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068596 5076 flags.go:64] FLAG: --cloud-provider="external" Nov 02 22:52:31.073851 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068600 5076 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 22:52:31.073851 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068604 5076 flags.go:64] FLAG: --cluster-domain="" Nov 02 22:52:31.074414 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068606 5076 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 22:52:31.074414 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068610 5076 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 22:52:31.074414 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068613 5076 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 22:52:31.074414 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068616 5076 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 22:52:31.074414 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068620 5076 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 22:52:31.074414 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068624 5076 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 22:52:31.074414 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068627 5076 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 22:52:31.074414 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068630 5076 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 22:52:31.074414 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068634 5076 flags.go:64] FLAG: --contention-profiling="false" Nov 02 22:52:31.074414 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068638 5076 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 22:52:31.074414 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068641 5076 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 22:52:31.074414 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068644 5076 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 22:52:31.074414 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068648 5076 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 22:52:31.074414 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068652 5076 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 22:52:31.074414 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068655 5076 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 22:52:31.074414 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068658 5076 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 22:52:31.074414 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068661 5076 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 22:52:31.074414 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068664 5076 flags.go:64] FLAG: --enable-server="true" Nov 02 22:52:31.074414 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068667 5076 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 22:52:31.074414 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068672 5076 flags.go:64] FLAG: --event-burst="100" Nov 02 22:52:31.074414 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068675 5076 flags.go:64] FLAG: --event-qps="50" Nov 02 22:52:31.074414 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068678 5076 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 22:52:31.074414 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068681 5076 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 22:52:31.074414 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068685 5076 flags.go:64] FLAG: --eviction-hard="" Nov 02 22:52:31.075010 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068690 5076 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 22:52:31.075010 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068693 5076 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 22:52:31.075010 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068696 5076 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 22:52:31.075010 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068699 5076 flags.go:64] FLAG: --eviction-soft="" Nov 02 22:52:31.075010 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068703 5076 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 22:52:31.075010 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068706 5076 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 22:52:31.075010 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068708 5076 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 22:52:31.075010 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068711 5076 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 22:52:31.075010 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068714 5076 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 22:52:31.075010 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068717 5076 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 22:52:31.075010 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068720 5076 flags.go:64] FLAG: --feature-gates="" Nov 02 22:52:31.075010 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068724 5076 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 22:52:31.075010 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068728 5076 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 22:52:31.075010 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068732 5076 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 22:52:31.075010 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068736 5076 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 22:52:31.075010 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068739 5076 flags.go:64] FLAG: --healthz-port="10248" Nov 02 22:52:31.075010 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068742 5076 flags.go:64] FLAG: --help="false" Nov 02 22:52:31.075010 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068745 5076 flags.go:64] FLAG: --hostname-override="ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:52:31.075010 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068748 5076 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 22:52:31.075010 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068751 5076 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 22:52:31.075010 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068754 5076 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 22:52:31.075010 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068758 5076 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 22:52:31.075010 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068761 5076 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 22:52:31.075575 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068764 5076 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 22:52:31.075575 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068767 5076 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 22:52:31.075575 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068770 5076 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 22:52:31.075575 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068773 5076 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 22:52:31.075575 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068776 5076 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 22:52:31.075575 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068780 5076 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 22:52:31.075575 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068783 5076 flags.go:64] FLAG: --kube-reserved="" Nov 02 22:52:31.075575 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068786 5076 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 22:52:31.075575 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068789 5076 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 22:52:31.075575 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068797 5076 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 22:52:31.075575 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068801 5076 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 22:52:31.075575 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068803 5076 flags.go:64] FLAG: --lock-file="" Nov 02 22:52:31.075575 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068806 5076 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 22:52:31.075575 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068809 5076 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 22:52:31.075575 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068812 5076 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 22:52:31.075575 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068822 5076 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 22:52:31.075575 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068825 5076 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 22:52:31.075575 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068828 5076 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 22:52:31.075575 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068831 5076 flags.go:64] FLAG: --logging-format="text" Nov 02 22:52:31.075575 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068834 5076 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 22:52:31.075575 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068838 5076 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 22:52:31.075575 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068846 5076 flags.go:64] FLAG: --manifest-url="" Nov 02 22:52:31.075575 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068849 5076 flags.go:64] FLAG: --manifest-url-header="" Nov 02 22:52:31.075575 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068854 5076 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 22:52:31.075575 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068857 5076 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 22:52:31.076271 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068860 5076 flags.go:64] FLAG: --max-pods="110" Nov 02 22:52:31.076271 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068863 5076 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 22:52:31.076271 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068866 5076 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 22:52:31.076271 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068869 5076 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 22:52:31.076271 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068872 5076 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 22:52:31.076271 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068875 5076 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 22:52:31.076271 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068878 5076 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 22:52:31.076271 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068881 5076 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 22:52:31.076271 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068907 5076 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 22:52:31.076271 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068910 5076 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 22:52:31.076271 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068913 5076 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 22:52:31.076271 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068916 5076 flags.go:64] FLAG: --pod-cidr="" Nov 02 22:52:31.076271 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068919 5076 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 22:52:31.076271 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068922 5076 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 22:52:31.076271 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068925 5076 flags.go:64] FLAG: --pods-per-core="0" Nov 02 22:52:31.076271 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068928 5076 flags.go:64] FLAG: --port="10250" Nov 02 22:52:31.076271 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068936 5076 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 22:52:31.076271 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068945 5076 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-06dbd8718a6f823b3" Nov 02 22:52:31.076271 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068949 5076 flags.go:64] FLAG: --qos-reserved="" Nov 02 22:52:31.076271 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068952 5076 flags.go:64] FLAG: --read-only-port="10255" Nov 02 22:52:31.076271 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068955 5076 flags.go:64] FLAG: --register-node="true" Nov 02 22:52:31.076271 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068958 5076 flags.go:64] FLAG: --register-with-taints="" Nov 02 22:52:31.076271 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068965 5076 flags.go:64] FLAG: --registry-burst="10" Nov 02 22:52:31.076271 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068968 5076 flags.go:64] FLAG: --registry-qps="5" Nov 02 22:52:31.076271 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068971 5076 flags.go:64] FLAG: --reserved-cpus="" Nov 02 22:52:31.076271 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068975 5076 flags.go:64] FLAG: --reserved-memory="" Nov 02 22:52:31.076885 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068979 5076 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 22:52:31.076885 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068982 5076 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 22:52:31.076885 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068987 5076 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 22:52:31.076885 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068990 5076 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 22:52:31.076885 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068993 5076 flags.go:64] FLAG: --runonce="false" Nov 02 22:52:31.076885 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068996 5076 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 22:52:31.076885 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.068999 5076 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 22:52:31.076885 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.069002 5076 flags.go:64] FLAG: --seccomp-default="false" Nov 02 22:52:31.076885 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.069005 5076 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 22:52:31.076885 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.069008 5076 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 22:52:31.076885 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.069011 5076 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 22:52:31.076885 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.069015 5076 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 22:52:31.076885 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.069018 5076 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 22:52:31.076885 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.069021 5076 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 22:52:31.076885 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.069024 5076 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 22:52:31.076885 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.069027 5076 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 22:52:31.076885 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.069030 5076 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 22:52:31.076885 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.069033 5076 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 22:52:31.076885 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.069036 5076 flags.go:64] FLAG: --system-cgroups="" Nov 02 22:52:31.076885 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.069039 5076 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 22:52:31.076885 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.069044 5076 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 22:52:31.076885 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.069047 5076 flags.go:64] FLAG: --tls-cert-file="" Nov 02 22:52:31.076885 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.069050 5076 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 22:52:31.076885 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.069056 5076 flags.go:64] FLAG: --tls-min-version="" Nov 02 22:52:31.076885 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.069060 5076 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 22:52:31.077537 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.069063 5076 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 22:52:31.077537 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.069065 5076 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 22:52:31.077537 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.069068 5076 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 22:52:31.077537 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.069071 5076 flags.go:64] FLAG: --v="2" Nov 02 22:52:31.077537 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.069076 5076 flags.go:64] FLAG: --version="false" Nov 02 22:52:31.077537 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.069080 5076 flags.go:64] FLAG: --vmodule="" Nov 02 22:52:31.077537 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.069084 5076 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 22:52:31.077537 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.069088 5076 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 22:52:31.077537 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069228 5076 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:52:31.077537 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069232 5076 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:52:31.077537 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069235 5076 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:52:31.077537 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069238 5076 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:52:31.077537 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069241 5076 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:52:31.077537 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069244 5076 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:52:31.077537 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069246 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:52:31.077537 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069249 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:52:31.077537 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069251 5076 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:52:31.077537 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069254 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:52:31.077537 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069257 5076 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:52:31.077537 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069259 5076 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:52:31.077537 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069262 5076 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:52:31.078056 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069265 5076 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:52:31.078056 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069267 5076 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:52:31.078056 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069270 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:52:31.078056 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069272 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:52:31.078056 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069275 5076 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:52:31.078056 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069277 5076 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:52:31.078056 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069280 5076 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:52:31.078056 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069282 5076 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:52:31.078056 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069288 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:52:31.078056 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069290 5076 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:52:31.078056 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069293 5076 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:52:31.078056 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069296 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:52:31.078056 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069299 5076 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:52:31.078056 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069302 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:52:31.078056 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069304 5076 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:52:31.078056 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069307 5076 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:52:31.078056 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069309 5076 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:52:31.078056 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069312 5076 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:52:31.078056 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069314 5076 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:52:31.078056 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069317 5076 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:52:31.078709 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069320 5076 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:52:31.078709 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069324 5076 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:52:31.078709 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069327 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:52:31.078709 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069331 5076 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:52:31.078709 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069335 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:52:31.078709 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069338 5076 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:52:31.078709 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069340 5076 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:52:31.078709 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069343 5076 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:52:31.078709 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069346 5076 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:52:31.078709 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069348 5076 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:52:31.078709 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069351 5076 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:52:31.078709 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069353 5076 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:52:31.078709 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069356 5076 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:52:31.078709 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069358 5076 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:52:31.078709 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069361 5076 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:52:31.078709 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069364 5076 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:52:31.078709 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069366 5076 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:52:31.078709 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069369 5076 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:52:31.078709 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069371 5076 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:52:31.079503 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069374 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:52:31.079503 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069378 5076 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:52:31.079503 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069381 5076 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:52:31.079503 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069383 5076 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:52:31.079503 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069386 5076 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:52:31.079503 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069389 5076 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:52:31.079503 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069392 5076 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:52:31.079503 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069394 5076 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:52:31.079503 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069397 5076 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:52:31.079503 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069399 5076 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:52:31.079503 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069402 5076 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:52:31.079503 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069405 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:52:31.079503 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069407 5076 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:52:31.079503 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069410 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:52:31.079503 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069413 5076 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:52:31.079503 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069415 5076 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:52:31.079503 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069418 5076 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:52:31.079503 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069421 5076 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:52:31.079503 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069424 5076 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:52:31.079999 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069426 5076 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:52:31.079999 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069429 5076 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:52:31.079999 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069432 5076 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:52:31.079999 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069435 5076 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:52:31.079999 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069438 5076 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:52:31.079999 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069440 5076 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:52:31.079999 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069443 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:52:31.079999 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069445 5076 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:52:31.079999 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069449 5076 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:52:31.079999 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069453 5076 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:52:31.079999 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069457 5076 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:52:31.079999 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069459 5076 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:52:31.079999 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069462 5076 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:52:31.079999 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069465 5076 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:52:31.079999 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069470 5076 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:52:31.079999 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069472 5076 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:52:31.079999 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069475 5076 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:52:31.079999 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069478 5076 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:52:31.079999 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069481 5076 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:52:31.080721 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069489 5076 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:52:31.080721 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069492 5076 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:52:31.080721 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069495 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:52:31.080721 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069498 5076 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:52:31.080721 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069500 5076 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:52:31.080721 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069503 5076 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:52:31.080721 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069505 5076 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:52:31.080721 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069508 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:52:31.080721 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069511 5076 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:52:31.080721 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069513 5076 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:52:31.080721 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069516 5076 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:52:31.080721 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069518 5076 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:52:31.080721 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069521 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:52:31.080721 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069525 5076 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:52:31.080721 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069528 5076 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:52:31.080721 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069531 5076 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:52:31.081408 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.069558 5076 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:52:31.081408 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069603 5076 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:52:31.081408 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069608 5076 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:52:31.081408 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069611 5076 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:52:31.081408 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069615 5076 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:52:31.081408 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069618 5076 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:52:31.081408 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069621 5076 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:52:31.081408 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069623 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:52:31.081408 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069626 5076 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:52:31.081408 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069630 5076 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:52:31.081408 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069634 5076 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:52:31.081408 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069638 5076 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:52:31.081408 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069642 5076 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:52:31.081408 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069645 5076 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:52:31.081408 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069648 5076 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:52:31.081881 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069650 5076 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:52:31.081881 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069653 5076 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:52:31.081881 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069661 5076 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:52:31.081881 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069664 5076 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:52:31.081881 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069667 5076 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:52:31.081881 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069669 5076 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:52:31.081881 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069672 5076 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:52:31.081881 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069675 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:52:31.081881 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069677 5076 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:52:31.081881 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069680 5076 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:52:31.081881 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069683 5076 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:52:31.081881 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069685 5076 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:52:31.081881 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069688 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:52:31.081881 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069690 5076 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:52:31.081881 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069694 5076 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:52:31.081881 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069697 5076 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:52:31.081881 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069700 5076 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:52:31.081881 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069702 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:52:31.081881 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069705 5076 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:52:31.081881 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069707 5076 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:52:31.082539 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069710 5076 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:52:31.082539 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069712 5076 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:52:31.082539 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069715 5076 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:52:31.082539 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069718 5076 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:52:31.082539 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069721 5076 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:52:31.082539 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069724 5076 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:52:31.082539 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069727 5076 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:52:31.082539 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069729 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:52:31.082539 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069732 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:52:31.082539 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069736 5076 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:52:31.082539 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069739 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:52:31.082539 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069742 5076 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:52:31.082539 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069744 5076 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:52:31.082539 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069747 5076 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:52:31.082539 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069750 5076 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:52:31.082539 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069758 5076 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:52:31.082539 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069761 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:52:31.082539 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069763 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:52:31.082539 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069766 5076 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:52:31.082539 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069769 5076 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:52:31.083130 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069771 5076 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:52:31.083130 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069774 5076 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:52:31.083130 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069776 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:52:31.083130 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069779 5076 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:52:31.083130 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069782 5076 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:52:31.083130 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069784 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:52:31.083130 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069788 5076 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:52:31.083130 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069791 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:52:31.083130 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069793 5076 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:52:31.083130 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069796 5076 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:52:31.083130 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069798 5076 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:52:31.083130 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069800 5076 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:52:31.083130 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069803 5076 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:52:31.083130 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069806 5076 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:52:31.083130 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069808 5076 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:52:31.083130 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069812 5076 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:52:31.083130 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069815 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:52:31.083130 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069820 5076 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:52:31.083130 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069822 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:52:31.083737 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069825 5076 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:52:31.083737 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069828 5076 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:52:31.083737 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069832 5076 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:52:31.083737 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069835 5076 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:52:31.083737 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069838 5076 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:52:31.083737 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069840 5076 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:52:31.083737 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069843 5076 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:52:31.083737 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069846 5076 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:52:31.083737 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069848 5076 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:52:31.083737 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069857 5076 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:52:31.083737 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069860 5076 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:52:31.083737 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069862 5076 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:52:31.083737 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069865 5076 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:52:31.083737 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069867 5076 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:52:31.083737 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069870 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:52:31.083737 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069873 5076 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:52:31.083737 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069875 5076 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:52:31.083737 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069878 5076 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:52:31.083737 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069880 5076 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:52:31.084311 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069885 5076 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:52:31.084311 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069901 5076 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:52:31.084311 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069905 5076 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:52:31.084311 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069907 5076 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:52:31.084311 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069910 5076 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:52:31.084311 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069912 5076 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:52:31.084311 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069915 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:52:31.084311 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069918 5076 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:52:31.084311 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069921 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:52:31.084311 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069924 5076 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:52:31.084311 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069927 5076 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:52:31.084311 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069929 5076 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:52:31.084311 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069932 5076 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:52:31.084311 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.069937 5076 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:52:31.084311 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.073712 5076 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 22:52:31.084311 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.073730 5076 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 22:52:31.084311 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.073748 5076 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 22:52:31.084311 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.073753 5076 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 22:52:31.084311 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073808 5076 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:52:31.084796 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073814 5076 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:52:31.084796 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073818 5076 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:52:31.084796 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073821 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:52:31.084796 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073824 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:52:31.084796 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073827 5076 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:52:31.084796 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073830 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:52:31.084796 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073833 5076 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:52:31.084796 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073835 5076 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:52:31.084796 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073839 5076 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:52:31.084796 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073844 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:52:31.084796 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073847 5076 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:52:31.084796 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073850 5076 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:52:31.084796 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073853 5076 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:52:31.084796 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073856 5076 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:52:31.084796 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073859 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:52:31.084796 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073861 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:52:31.084796 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073864 5076 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:52:31.084796 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073867 5076 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:52:31.084796 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073870 5076 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:52:31.085333 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073873 5076 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:52:31.085333 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073876 5076 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:52:31.085333 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073878 5076 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:52:31.085333 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073881 5076 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:52:31.085333 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073883 5076 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:52:31.085333 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073903 5076 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:52:31.085333 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073906 5076 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:52:31.085333 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073909 5076 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:52:31.085333 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073913 5076 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:52:31.085333 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073919 5076 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:52:31.085333 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073923 5076 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:52:31.085333 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073926 5076 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:52:31.085333 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073929 5076 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:52:31.085333 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073932 5076 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:52:31.085333 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073934 5076 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:52:31.085333 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073937 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:52:31.085333 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073940 5076 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:52:31.085333 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073942 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:52:31.085333 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073945 5076 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:52:31.085782 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073948 5076 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:52:31.085782 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073950 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:52:31.085782 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073952 5076 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:52:31.085782 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073955 5076 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:52:31.085782 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073958 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:52:31.085782 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073961 5076 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:52:31.085782 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073963 5076 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:52:31.085782 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073966 5076 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:52:31.085782 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073969 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:52:31.085782 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073971 5076 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:52:31.085782 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073974 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:52:31.085782 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073977 5076 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:52:31.085782 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073980 5076 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:52:31.085782 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073983 5076 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:52:31.085782 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073986 5076 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:52:31.085782 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073988 5076 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:52:31.085782 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073991 5076 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:52:31.085782 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073994 5076 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:52:31.085782 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.073997 5076 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:52:31.086268 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074000 5076 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:52:31.086268 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074002 5076 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:52:31.086268 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074005 5076 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:52:31.086268 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074008 5076 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:52:31.086268 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074011 5076 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:52:31.086268 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074014 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:52:31.086268 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074017 5076 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:52:31.086268 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074020 5076 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:52:31.086268 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074022 5076 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:52:31.086268 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074025 5076 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:52:31.086268 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074027 5076 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:52:31.086268 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074030 5076 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:52:31.086268 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074033 5076 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:52:31.086268 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074035 5076 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:52:31.086268 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074038 5076 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:52:31.086268 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074040 5076 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:52:31.086268 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074043 5076 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:52:31.086268 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074046 5076 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:52:31.086268 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074048 5076 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:52:31.086268 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074051 5076 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:52:31.086772 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074054 5076 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:52:31.086772 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074057 5076 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:52:31.086772 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074060 5076 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:52:31.086772 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074063 5076 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:52:31.086772 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074065 5076 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:52:31.086772 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074068 5076 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:52:31.086772 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074070 5076 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:52:31.086772 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074073 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:52:31.086772 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074075 5076 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:52:31.086772 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074078 5076 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:52:31.086772 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074080 5076 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:52:31.086772 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074083 5076 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:52:31.086772 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074086 5076 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:52:31.086772 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074089 5076 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:52:31.086772 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074091 5076 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:52:31.086772 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074094 5076 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:52:31.086772 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074097 5076 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:52:31.086772 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074100 5076 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:52:31.086772 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074103 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:52:31.086772 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074106 5076 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:52:31.087267 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074108 5076 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:52:31.087267 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074111 5076 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:52:31.087267 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074113 5076 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:52:31.087267 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074116 5076 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:52:31.087267 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074119 5076 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:52:31.087267 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074122 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:52:31.087267 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074124 5076 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:52:31.087267 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074127 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:52:31.087267 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.074155 5076 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:52:31.087267 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074297 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:52:31.087267 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074301 5076 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:52:31.087267 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074304 5076 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:52:31.087267 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074307 5076 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:52:31.087267 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074309 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:52:31.087267 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074312 5076 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:52:31.087267 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074315 5076 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:52:31.087651 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074317 5076 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:52:31.087651 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074320 5076 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:52:31.087651 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074323 5076 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:52:31.087651 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074325 5076 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:52:31.087651 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074328 5076 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:52:31.087651 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074330 5076 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:52:31.087651 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074333 5076 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:52:31.087651 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074336 5076 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:52:31.087651 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074338 5076 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:52:31.087651 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074341 5076 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:52:31.087651 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074343 5076 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:52:31.087651 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074346 5076 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:52:31.087651 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074349 5076 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:52:31.087651 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074352 5076 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:52:31.087651 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074360 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:52:31.087651 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074364 5076 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:52:31.087651 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074366 5076 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:52:31.087651 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074369 5076 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:52:31.087651 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074371 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:52:31.087651 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074374 5076 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:52:31.088145 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074377 5076 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:52:31.088145 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074379 5076 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:52:31.088145 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074382 5076 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:52:31.088145 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074385 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:52:31.088145 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074387 5076 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:52:31.088145 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074390 5076 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:52:31.088145 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074392 5076 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:52:31.088145 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074395 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:52:31.088145 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074397 5076 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:52:31.088145 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074400 5076 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:52:31.088145 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074402 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:52:31.088145 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074405 5076 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:52:31.088145 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074407 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:52:31.088145 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074410 5076 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:52:31.088145 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074413 5076 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:52:31.088145 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074415 5076 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:52:31.088145 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074418 5076 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:52:31.088145 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074422 5076 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:52:31.088145 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074426 5076 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:52:31.088589 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074429 5076 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:52:31.088589 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074431 5076 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:52:31.088589 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074434 5076 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:52:31.088589 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074437 5076 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:52:31.088589 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074439 5076 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:52:31.088589 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074442 5076 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:52:31.088589 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074444 5076 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:52:31.088589 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074448 5076 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:52:31.088589 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074452 5076 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:52:31.088589 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074455 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:52:31.088589 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074458 5076 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:52:31.088589 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074461 5076 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:52:31.088589 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074464 5076 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:52:31.088589 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074466 5076 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:52:31.088589 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074468 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:52:31.088589 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074471 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:52:31.088589 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074477 5076 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:52:31.088589 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074479 5076 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:52:31.088589 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074482 5076 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:52:31.089141 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074484 5076 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:52:31.089141 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074487 5076 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:52:31.089141 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074490 5076 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:52:31.089141 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074492 5076 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:52:31.089141 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074495 5076 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:52:31.089141 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074497 5076 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:52:31.089141 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074500 5076 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:52:31.089141 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074503 5076 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:52:31.089141 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074505 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:52:31.089141 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074508 5076 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:52:31.089141 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074510 5076 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:52:31.089141 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074513 5076 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:52:31.089141 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074516 5076 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:52:31.089141 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074518 5076 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:52:31.089141 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074521 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:52:31.089141 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074524 5076 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:52:31.089141 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074527 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:52:31.089141 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074529 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:52:31.089141 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074532 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:52:31.089644 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074534 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:52:31.089644 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074537 5076 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:52:31.089644 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074539 5076 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:52:31.089644 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074542 5076 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:52:31.089644 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074545 5076 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:52:31.089644 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074548 5076 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:52:31.089644 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074550 5076 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:52:31.089644 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074553 5076 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:52:31.089644 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074555 5076 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:52:31.089644 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074558 5076 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:52:31.089644 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074562 5076 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:52:31.089644 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074564 5076 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:52:31.089644 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074567 5076 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:52:31.089644 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074570 5076 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:52:31.089644 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074573 5076 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:52:31.089644 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074575 5076 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:52:31.089644 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074578 5076 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:52:31.089644 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074580 5076 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:52:31.089644 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074583 5076 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:52:31.089644 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074586 5076 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:52:31.090318 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074589 5076 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:52:31.090318 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074591 5076 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:52:31.090318 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.074614 5076 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:52:31.090318 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074645 5076 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:52:31.090318 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074649 5076 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:52:31.090318 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074652 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:52:31.090318 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074655 5076 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:52:31.090318 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074657 5076 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:52:31.090318 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074660 5076 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:52:31.090318 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074665 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:52:31.090318 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074668 5076 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:52:31.090318 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074671 5076 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:52:31.090318 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074674 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:52:31.090318 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074676 5076 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:52:31.090318 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074679 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:52:31.090318 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074681 5076 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:52:31.090834 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074684 5076 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:52:31.090834 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074687 5076 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:52:31.090834 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074690 5076 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:52:31.090834 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074694 5076 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:52:31.090834 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074698 5076 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:52:31.090834 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074700 5076 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:52:31.090834 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074703 5076 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:52:31.090834 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074707 5076 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:52:31.090834 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074710 5076 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:52:31.090834 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074713 5076 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:52:31.090834 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074715 5076 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:52:31.090834 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074718 5076 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:52:31.090834 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074722 5076 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:52:31.090834 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074726 5076 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:52:31.090834 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074729 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:52:31.090834 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074732 5076 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:52:31.090834 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074735 5076 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:52:31.090834 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074738 5076 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:52:31.090834 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074741 5076 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:52:31.091420 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074744 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:52:31.091420 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074746 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:52:31.091420 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074749 5076 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:52:31.091420 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074751 5076 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:52:31.091420 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074754 5076 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:52:31.091420 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074757 5076 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:52:31.091420 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074760 5076 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:52:31.091420 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074763 5076 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:52:31.091420 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074765 5076 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:52:31.091420 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074768 5076 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:52:31.091420 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074771 5076 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:52:31.091420 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074773 5076 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:52:31.091420 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074776 5076 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:52:31.091420 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074778 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:52:31.091420 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074782 5076 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:52:31.091420 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074785 5076 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:52:31.091420 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074787 5076 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:52:31.091420 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074790 5076 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:52:31.091420 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074792 5076 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:52:31.091420 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074795 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:52:31.092191 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074800 5076 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:52:31.092191 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074802 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:52:31.092191 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074805 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:52:31.092191 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074807 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:52:31.092191 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074810 5076 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:52:31.092191 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074812 5076 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:52:31.092191 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074815 5076 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:52:31.092191 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074818 5076 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:52:31.092191 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074820 5076 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:52:31.092191 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074823 5076 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:52:31.092191 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074825 5076 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:52:31.092191 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074828 5076 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:52:31.092191 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074831 5076 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:52:31.092191 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074834 5076 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:52:31.092191 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074836 5076 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:52:31.092191 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074839 5076 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:52:31.092191 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074841 5076 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:52:31.092191 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074844 5076 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:52:31.092191 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074848 5076 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:52:31.092852 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074851 5076 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:52:31.092852 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074854 5076 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:52:31.092852 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074856 5076 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:52:31.092852 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074859 5076 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:52:31.092852 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074861 5076 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:52:31.092852 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074864 5076 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:52:31.092852 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074866 5076 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:52:31.092852 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074869 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:52:31.092852 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074872 5076 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:52:31.092852 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074875 5076 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:52:31.092852 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074877 5076 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:52:31.092852 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074880 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:52:31.092852 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074883 5076 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:52:31.092852 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074898 5076 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:52:31.092852 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074901 5076 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:52:31.092852 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074904 5076 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:52:31.092852 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074906 5076 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:52:31.092852 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074909 5076 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:52:31.092852 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074912 5076 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:52:31.092852 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074914 5076 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:52:31.093558 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074917 5076 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:52:31.093558 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074921 5076 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:52:31.093558 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074924 5076 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:52:31.093558 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074926 5076 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:52:31.093558 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074929 5076 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:52:31.093558 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074932 5076 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:52:31.093558 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074934 5076 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:52:31.093558 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074937 5076 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:52:31.093558 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074939 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:52:31.093558 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074942 5076 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:52:31.093558 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074944 5076 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:52:31.093558 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074949 5076 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:52:31.093558 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074951 5076 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:52:31.093558 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074954 5076 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:52:31.093558 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:31.074957 5076 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:52:31.093558 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.075017 5076 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 22:52:31.093558 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.076023 5076 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 22:52:31.093558 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.076095 5076 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 22:52:31.093558 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.076301 5076 server.go:1014] "Starting client certificate rotation" Nov 02 22:52:31.094037 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.076449 5076 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 22:52:31.094037 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.076508 5076 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 22:46:48 +0000 UTC" deadline="2028-07-26 19:28:18.568045744 +0000 UTC" Nov 02 22:52:31.094037 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.076542 5076 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15164h35m47.491505179s" Nov 02 22:52:31.094037 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.076795 5076 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 22:52:31.094037 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.077853 5076 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 22:52:31.094037 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.079732 5076 log.go:25] "Validated CRI v1 runtime API" Nov 02 22:52:31.094037 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.083097 5076 log.go:25] "Validated CRI v1 image API" Nov 02 22:52:31.094037 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.083922 5076 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 22:52:31.094037 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.085929 5076 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ab63f09b-d9ce-4d89-a1e7-44b7db89c253:/dev/nvme0n1p3 b143f1c5-992e-460f-80d5-80a31569b717:/dev/nvme0n1p4] Nov 02 22:52:31.094320 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.085947 5076 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/055a42e181d047b2d0d0a3eb5266a6afecee615a52a63186bcdfc22ce49167c3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/055a42e181d047b2d0d0a3eb5266a6afecee615a52a63186bcdfc22ce49167c3/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/25637d3d0737f9e52181634fdb879f8dde56f0b2436d7b7e695a500d72cab169/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/25637d3d0737f9e52181634fdb879f8dde56f0b2436d7b7e695a500d72cab169/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/25ce3b94c1b208749a7a75bead39555de3944bf63e6d8f360ca695541ce8f157/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/25ce3b94c1b208749a7a75bead39555de3944bf63e6d8f360ca695541ce8f157/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/318c3a34f01581248a349ccfe0a15df08396d1a44a40b4c33f781061059a387e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/318c3a34f01581248a349ccfe0a15df08396d1a44a40b4c33f781061059a387e/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/31ea80300d42577ad470fb6f4391ef10c997d0836355f54625ad14cb535f4483/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/31ea80300d42577ad470fb6f4391ef10c997d0836355f54625ad14cb535f4483/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/387a216b6d58d9b749b3b5ca2cc899e1729c8ed618fd1fd4517f1cc35170c68e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/387a216b6d58d9b749b3b5ca2cc899e1729c8ed618fd1fd4517f1cc35170c68e/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/47f55c937c827013ee62eecf35fb78175c77ee2729e44dbf3f8d37893e51eb7d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/47f55c937c827013ee62eecf35fb78175c77ee2729e44dbf3f8d37893e51eb7d/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4f48f38b5e24171bd945b75111ee6fa5347a41c456770b64e8f7d43d1ee65e38/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4f48f38b5e24171bd945b75111ee6fa5347a41c456770b64e8f7d43d1ee65e38/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/54ad588bb8e86ed38f7a08e74739b501f3663a3a0f5cda683afea2c00d975db2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/54ad588bb8e86ed38f7a08e74739b501f3663a3a0f5cda683afea2c00d975db2/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/58436ee67e99f72594ebf75d78e03a97fbc28924da7c08d072582970e23d689e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/58436ee67e99f72594ebf75d78e03a97fbc28924da7c08d072582970e23d689e/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5d546ba5bc54172ecbea70ee0fbfebe48033ebf77bc7a94c95663240295e4292/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5d546ba5bc54172ecbea70ee0fbfebe48033ebf77bc7a94c95663240295e4292/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/696172ee14d01e9d35bdac289d3290e2adc725e4cafa322f60dc04e3cb208c47/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/696172ee14d01e9d35bdac289d3290e2adc725e4cafa322f60dc04e3cb208c47/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/87ead83b47d5f3fc7380e32329e4f44fd3f61c3f216506076b57c9c75269f0d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/87ead83b47d5f3fc7380e32329e4f44fd3f61c3f216506076b57c9c75269f0d5/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/914d69658f5c63ff96087f02576c562a6e1167cc1722fef6a93a4c62118ab3ab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/914d69658f5c63ff96087f02576c562a6e1167cc1722fef6a93a4c62118ab3ab/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/923bdca363bdecec9681204d8c7d61a686ad35aad15b349b72375be1ca7ac71e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/923bdca363bdecec9681204d8c7d61a686ad35aad15b349b72375be1ca7ac71e/userdata/shm major:0 minor:338 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/924af956a0c2c62f133a420c317fb5dd6c2d5bbfde22749bfceb02dc429d5b67/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/924af956a0c2c62f133a420c317fb5dd6c2d5bbfde22749bfceb02dc429d5b67/userdata/shm major:0 minor:330 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9332cac3ad414a7a953cd8f44d53bed498e0df65d2ec5b79aae1ba771af997e6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9332cac3ad414a7a953cd8f44d53bed498e0df65d2ec5b79aae1ba771af997e6/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ef782737103253b57fdf6f5c12cb642b00ca83215d5a52e554ce2f17e691f49/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ef782737103253b57fdf6f5c12cb642b00ca83215d5a52e554ce2f17e691f49/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a7eef4c62c90023e339d07ef34aa6a936384058d68431afa17a67efc0579b03e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a7eef4c62c90023e339d07ef34aa6a936384058d68431afa17a67efc0579b03e/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b1a2da6bba00ba8c0594bb3dbbeb3cca05136ff2fdbe870e87d5d1cbbfa6254a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b1a2da6bba00ba8c0594bb3dbbeb3cca05136ff2fdbe870e87d5d1cbbfa6254a/userdata/shm major:0 minor:245 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bae187d60c71e81ac2558a9049659dfeb1817e080bfab2d74e4c2e79695056ee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bae187d60c71e81ac2558a9049659dfeb1817e080bfab2d74e4c2e79695056ee/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/de924197b3afd07e8c6b2d9ab15d26217b20b69c974fcc935146dd25598040e8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/de924197b3afd07e8c6b2d9ab15d26217b20b69c974fcc935146dd25598040e8/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eb0cdddb2bd55368782469edc79d2349ea253148a8e8af8a37591b6f4181ba61/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eb0cdddb2bd55368782469edc79d2349ea253148a8e8af8a37591b6f4181ba61/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/edff1bb6209a6e5782a55a8157d2c1884fef8ac84da3480932b04f9c9ebebcec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/edff1bb6209a6e5782a55a8157d2c1884fef8ac84da3480932b04f9c9ebebcec/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ee0f5d8810757aa96638f8b91b9e18c1185af1ff8cb70d801bfedb7b95ddda50/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ee0f5d8810757aa96638f8b91b9e18c1185af1ff8cb70d801bfedb7b95ddda50/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f38c14908ece618cc8f271a7b83d2be46b29f52d2bcf1ca5db920bf6436246f8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f38c14908ece618cc8f271a7b83d2be46b29f52d2bcf1ca5db920bf6436246f8/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/00b95c29-7df8-4602-bd0b-06b6281d5dc1/volumes/kubernetes.io~projected/kube-api-access-zt847:{mountpoint:/var/lib/kubelet/pods/00b95c29-7df8-4602-bd0b-06b6281d5dc1/volumes/kubernetes.io~projected/kube-api-access-zt847 major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0237eeb1-8c85-475e-8580-6dec324de705/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/0237eeb1-8c85-475e-8580-6dec324de705/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0237eeb1-8c85-475e-8580-6dec324de705/volumes/kubernetes.io~projected/kube-api-access-nwnxj:{mountpoint:/var/lib/kubelet/pods/0237eeb1-8c85-475e-8580-6dec324de705/volumes/kubernetes.io~projected/kube-api-access-nwnxj major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0237eeb1-8c85-475e-8580-6dec324de705/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/0237eeb1-8c85-475e-8580-6dec324de705/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0237eeb1-8c85-475e-8580-6dec324de705/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/0237eeb1-8c85-475e-8580-6dec324de705/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b6b45c3-5275-4877-a84b-15e602100c40/volumes/kubernetes.io~projected/kube-api-access-6flqc:{mountpoint:/var/lib/kubelet/pods/0b6b45c3-5275-4877-a84b-15e602100c40/volumes/kubernetes.io~projected/kube-api-access-6flqc major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d008a27-d5ba-49cb-b3fb-abdecd723abd/volumes/kubernetes.io~projected/kube-api-access-fqdk5:{mountpoint:/var/lib/kubelet/pods/0d008a27-d5ba-49cb-b3fb-abdecd723abd/volumes/kubernetes.io~projected/kube-api-access-fqdk5 major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/105406c8-6800-4562-9651-962bee9f8d73/volumes/kubernetes.io~projected/kube-api-access-9f4r4:{mountpoint:/var/lib/kubelet/pods/105406c8-6800-4562-9651-962bee9f8d73/volumes/kubernetes.io~projected/kube-api-access-9f4r4 major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15789a0d-e030-45e0-a129-41de738ef767/volumes/kubernetes.io~projected/kube-api-access-jnt9q:{mountpoint:/var/lib/kubelet/pods/15789a0d-e030-45e0-a129-41de738ef767/volumes/kubernetes.io~projected/kube-api-access-jnt9q major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/236450f1-d9fa-488b-9022-ab4f5517549f/volumes/kubernetes.io~projected/kube-api-access-vxxqc:{mountpoint:/var/lib/kubelet/pods/236450f1-d9fa-488b-9022-ab4f5517549f/volumes/kubernetes.io~projected/kube-api-access-vxxqc major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49/volumes/kubernetes.io~projected/kube-api-access-kkqkc:{mountpoint:/var/lib/kubelet/pods/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49/volumes/kubernetes.io~projected/kube-api-access-kkqkc major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a464bf3-77fa-4fcd-8593-327fa1f4a19a/volumes/kubernetes.io~projected/kube-api-access-cml47:{mountpoint:/var/lib/kubelet/pods/3a464bf3-77fa-4fcd-8593-327fa1f4a19a/volumes/kubernetes.io~projected/kube-api-access-cml47 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42d598be-a6f1-486e-9ac3-2f5ece575f89/volumes/kubernetes.io~projected/kube-api-access-stv25:{mountpoint:/var/lib/kubelet/pods/42d598be-a6f1-486e-9ac3-2f5ece575f89/volumes/kubernetes.io~projected/kube-api-access-stv25 major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46eae5da-cf0b-4663-bdeb-f8220fa08c05/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/46eae5da-cf0b-4663-bdeb-f8220fa08c05/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46eae5da-cf0b-4663-bdeb-f8220fa08c05/volumes/kubernetes.io~projected/kube-api-access-nkzvb:{mountpoint:/var/lib/kubelet/pods/46eae5da-cf0b-4663-bdeb-f8220fa08c05/volumes/kubernetes.io~projected/kube-api-access-nkzvb major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46eae5da-cf0b-4663-bdeb-f8220fa08c05/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/46eae5da-cf0b-4663-bdeb-f8220fa08c05/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46eae5da-cf0b-4663-bdeb-f8220fa08c05/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/46eae5da-cf0b-4663-bdeb-f8220fa08c05/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b9cd8c7-228b-451a-afe4-3af7a3bf0a42/volumes/kubernetes.io~projected/kube-api-access-ncmnm:{mountpoint:/var/lib/kubelet/pods/4b9cd8c7-228b-451a-afe4-3af7a3bf0a42/volumes/kubernetes.io~projected/kube-api-access-ncmnm major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f8b41ab-cbec-4f72-a5d9-853953748c0b/volumes/kubernetes.io~projected/kube-api-access-ctsxp:{mountpoint:/var/lib/kubelet/pods/4f8b41ab-cbec-4f72-a5d9-853953748c0b/volumes/kubernetes.io~projected/kube-api-access-ctsxp major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50d13ae8-4ab5-4ca7-a11a-ad632da19669/volumes/kubernetes.io~projected/kube-api-access-5j4hf:{mountpoint:/var/lib/kubelet/pods/50d13ae8-4ab5-4ca7-a11a-ad632da19669/volumes/kubernetes.io~projected/kube-api-access-5j4hf major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60832943-bf1f-4685-be00-08152c423b08/volumes/kubernetes.io~projected/kube-api-access-pmk5v:{mountpoint:/var/lib/kubelet/pods/60832943-bf1f-4685-be00-08152c423b08/volumes/kubernetes.io~projected/kube-api-access-pmk5v major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60832943-bf1f-4685-be00-08152c423b08/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/60832943-bf1f-4685-be00-08152c423b08/volumes/kubernetes.io~secret/serving-cert major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/668094b0-0e10-47aa-ba49-6944f7083bc6/volumes/kubernetes.io~projected/kube-api-access-k4shq:{mountpoint:/var/lib/kubelet/pods/668094b0-0e10-47aa-ba49-6944f7083bc6/volumes/kubernetes.io~projected/kube-api-access-k4shq major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b3753c7-7634-4209-8760-7a52da534549/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/6b3753c7-7634-4209-8760-7a52da534549/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b3753c7-7634-4209-8760-7a52da534549/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/6b3753c7-7634-4209-8760-7a52da534549/volumes/kubernetes.io~empty-dir/tmp major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b3753c7-7634-4209-8760-7a52da534549/volumes/kubernetes.io~projected/kube-api-access-wzxgm:{mountpoint:/var/lib/kubelet/pods/6b3753c7-7634-4209-8760-7a52da534549/volumes/kubernetes.io~projected/kube-api-access-wzxgm major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75f7d1d7-88e4-41e0-bdd3-a43df363b408/volumes/kubernetes.io~projected/kube-api-access-md7ml:{mountpoint:/var/lib/kubelet/pods/75f7d1d7-88e4-41e0-bdd3-a43df363b408/volumes/kubernetes.io~projected/kube-api-access-md7ml major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75f7d1d7-88e4-41e0-bdd3-a43df363b408/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/75f7d1d7-88e4-41e0-bdd3-a43df363b408/volumes/kubernetes.io~secret/default-certificate major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75f7d1d7-88e4-41e0-bdd3-a43df363b408/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/75f7d1d7-88e4-41e0-bdd3-a43df363b408/volumes/kubernetes.io~secret/stats-auth major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79e2a74e-667b-4851-9e8a-759cb2a91f98/volumes/kubernetes.io~projected/kube-api-access-lx46q:{mountpoint:/var/lib/kubelet/pods/79e2a74e-667b-4851-9e8a-759cb2a91f98/volumes/kubernetes.io~projected/kube-api-access-lx46q major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79e2a74e-667b-4851-9e8a-759cb2a91f98/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/79e2a74e-667b-4851-9e8a-759cb2a91f98/volumes/kubernetes.io~secret/signing-key major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fbcc658-b377-4da4-bf1d-e6274dfd805b/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/8fbcc658-b377-4da4-bf1d-e6274dfd805b/volumes/kubernetes.io~secret/agent-certs major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/volumes/kubernetes.io~projected/kube-api-access-89zw7:{mountpoint:/var/lib/kubelet/pods/9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/volumes/kubernetes.io~projected/kube-api-access-89zw7 major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/volumes/kubernetes.io~secret/serving-cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f98166f-8051-4dbd-9856-db692932050b/volumes/kubernetes.io~projected/kube-api-access-85k6p:{mountpoint:/var/lib/kubelet/pods/9f98166f-8051-4dbd-9856-db692932050b/volumes/kubernetes.io~projected/kube-api-access-85k6p major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac4a7fda-863e-4fb1-a98c-9b325a8c0df7/volumes/kubernetes.io~projected/kube-api-access-9t8ms:{mountpoint:/var/lib/kubelet/pods/ac4a7fda-863e-4fb1-a98c-9b325a8c0df7/volumes/kubernetes.io~projected/kube-api-access-9t8ms major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac4a7fda-863e-4fb1-a98c-9b325a8c0df7/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/ac4a7fda-863e-4fb1-a98c-9b325a8c0df7/volumes/kubernetes.io~secret/cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b858466f-3dcd-419c-9495-4dbb89e8d51b/volumes/kubernetes.io~projected/kube-api-access-ws5j7:{mountpoint:/var/lib/kubelet/pods/b858466f-3dcd-419c-9495-4dbb89e8d51b/volumes/kubernetes.io~projected/kube-api-access-ws5j7 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b858466f-3dcd-419c-9495-4dbb89e8d51b/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/b858466f-3dcd-419c-9495-4dbb89e8d51b/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb1743a1-6e7b-4f99-80e6-197a6916580d/volumes/kubernetes.io~projected/kube-api-access-6rt8s:{mountpoint:/var/lib/kubelet/pods/cb1743a1-6e7b-4f99-80e6-197a6916580d/volumes/kubernetes.io~projected/kube-api-access-6rt8s major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d12c211b-f607-43b9-b621-12f061f109d9/volumes/kubernetes.io~projected/kube-api-access-bx9mw:{mountpoint:/var/lib/kubelet/pods/d12c211b-f607-43b9-b621-12f061f109d9/volumes/kubernetes.io~projected/kube-api-access-bx9mw major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3575a78-aad1-41e0-bad4-63724365aeb1/volumes/kubernetes.io~projected/kube-api-access-vvg2k:{mountpoint:/var/lib/kubelet/pods/d3575a78-aad1-41e0-bad4-63724365aeb1/volumes/kubernetes.io~projected/kube-api-access-vvg2k major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df418d6f-b9d3-4019-967c-9511be123e1f/volumes/kubernetes.io~projected/kube-api-access-jjljw:{mountpoint:/var/lib/kubelet/pods/df418d6f-b9d3-4019-967c-9511be123e1f/volumes/kubernetes.io~projected/kube-api-access-jjljw major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df418d6f-b9d3-4019-967c-9511be123e1f/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/df418d6f-b9d3-4019-967c-9511be123e1f/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3a3bc57-6813-441c-bdc0-d39eb8214c61/volumes/kubernetes.io~projected/kube-api-access-q7nrp:{mountpoint:/var/lib/kubelet/pods/e3a3bc57-6813-441c-bdc0-d39eb8214c61/volumes/kubernetes.io~projected/kube-api-access-q7nrp major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee52c167-6361-4adb-9899-93ca4c2128b8/volumes/kubernetes.io~projected/kube-api-access-q7qr5:{mountpoint:/var/lib/kubelet/pods/ee52c167-6361-4adb-9899-93ca4c2128b8/volumes/kubernetes.io~projected/kube-api-access-q7qr5 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fbcab4d0-9dd6-4682-be01-c4f589697e95/volumes/kubernetes.io~projected/kube-api-access-24smc:{mountpoint:/var/lib/kubelet/pods/fbcab4d0-9dd6-4682-be01-c4f589697e95/volumes/kubernetes.io~projected/kube-api-access-24smc major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc69f07f-4001-4951-aff8-d3d6671cc5d4/volumes/kubernetes.io~projected/kube-api-access-n6jlv:{mountpoint:/var/lib/kubelet/pods/fc69f07f-4001-4951-aff8-d3d6671cc5d4/volumes/kubernetes.io~projected/kube-api-access-n6jlv major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd96552a-1a78-4fac-8faf-71d36f010905/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/fd96552a-1a78-4fac-8faf-71d36f010905/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:286 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/0fa7cb5a4473c1d22c601d5b116ff9dbc384dc0bf43f1a54de33cd566ebc4b1c/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/cba6d0e27d1622547b59eee3b0db6cbdd0cf6e742e71173e8945080a573377bd/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/b995b73f56003853e09d1be6d6beba8f1e7c4c99fc661273bf327d6cd438e91e/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/ec8aa72625f30db25a1f82800be9c81680aef91091bb2548a89dedc4525f8022/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/d8c14eee438cc25d2ba3c78539224dcc1dec70137c3be48eb97e6241240df8eb/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/620a80f3f1a278c23aeb03f26aafd5cffee898e58edbad882690afc4aeb51e3c/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/f566c046811a5e5bb31fcf4d917da9694a2221307f3779f58640bc538277223f/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/093a2767643522a648d5f7706739b3ed8ebd00f8cc4e419d86ba7ccb47754ef8/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/11eec1afb2b09b4da8a6dccdef2e23aa1ce696a755e7145fbd8e4b343287cdb1/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/3f0e7dbc7cba1d880df3d5d347cc526c6e4ad3d98273de81e812fce51767113e/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/fa54d8dda62bb48190138d3c592c77da590b155519330ba8030ab3e9f1abe607/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/3419256d627183503f4e385e0f42dc7f0057fca095ff0cc087cae474b80f51f5/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/9a8b34347f4487d2538ad9c371b6fa4a47e19e3fb95d2c8dd3b81232ed6f4f84/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/9fe678ef5533cc727701df35780d69e91eccb3584572e4182e492ce5441671d4/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/54d94b71e7cbdf4a46ab188eed09af7de1113f03a9b11aa3e259f7d14cb20d49/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/5a9ec92344eb8a31fb7126593f97c57305f09f71867161111548d7e6562d1216/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/4dc3429d5037f933155c387f67dd47cfe2edcc53e97a956b550775ac331d770d/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/589455f6bf20188c0469e60f2f30e7cc72ce18c53f4d8840dbd40357f9cfe639/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/54f2f96880e1218743436204518a4aa0897e3d3db3a6caf186475bc26c5609bb/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/1be6f9ebcc926c31a076d0940911d38d857a4b0d5867e75ace8fa8b8a66ab90d/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/d33d77163f87fc71089014cb0d7e767b1c8a467476fc9417226a3cc47fb8dbcf/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/9b88a897341b7f81cc6b05505a6c7ae1fb8255da6e0445a865a7a647c416989a/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/6782c56809cd45ffa61ea710e1cc35ee66113f0f493ea3dd60d3f2f07734b9dc/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/eca209a924fd1d6ecc4ae8e5b81f77223ef768a9a39333b97ecb825efc60c53b/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/8553eb8a34cd225e7e64e64a16ce34dcadeb6f94d707c4aa28ae97f29c341ab1/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/021014961e3ab4c0d72b8380f6a0af1d0637cba651483b903d3c9938980748a1/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/73903ebcfa96027e01069d7d665e3b4bce9927524559c787a86b599338ee8ac6/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/59bd7881f34ce1df264dde04fad81f6d238b4b5a9b1bb64d675fa2bbb13fbae7/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/1f1df66f6d36b31d455007352adfd1d847e13018c257d40aba2b9faa946d874b/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/3709b08c3ec8dc038c50c2775b17bb19307813ae34bf33d9abba274b5cff6c96/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/4af658208085b83479629093a148664915d5dd2368e0707433f6ec4164470d8f/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/2955b2712bb7093dc921a1befa0d2f359c675f7430028a4d454337080e205a4d/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/c2f5780255d96213f37956699e95b108a5572bf014c13fe80306b3eeb74aafff/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/d1ec609ca3637326c22474d29ebc26173b8ebc069b3daca1300079f9ca0ecdc3/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/e37f59bcb692b42fe0cfbc19ddfc07136a0c8478c688ea2112688a194e121764/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/67733a22a0168bfabd5817dc5406b7caaed4fd49346cc3c39ac0785fb6b5aee5/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/8a0048888e655fcb4cda19a57c086656b699d74cadb1a24a2340bb7025df1585/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/5504e480dc3bbe3325c608a022110f3e9ce0b866f1d6dc0e079b2bc72fe697fa/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/70c1aec4ce2770f972caf077602995b89db84738b00d90714042c62269489f84/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/207cf9d3370e32d4f5d68ad19531e604edc6e5494572f481fde64523b33f398e/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/711ade3097e88c01d273f9f6581f2a1aaad074d7259800956d216a41d0ad2466/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/c7c6cba4e8d061fb6706ec06a64fc4e6a38c5f94e3f6ad59baa8bdf94fcfa372/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/cd67d7f549f29d2d8c1da2c0764f7156f96382cc0a82af4517e6028b071d5758/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/c19dd4b454d6562b6dbc5e7f7960de64b21933ee4973d2fad22c31551a60d09c/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/ce0a01fb60f0b00fce7df40b02f3c04a8cee2857ab97afc14020dda33cb9adcb/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/50c5d24b2966ebe58d5d30901ae3e3867aa659c4da811b025261951d0dfaeee3/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/1d729b1dcd4c1c6b24475c6b7d2d7e6a9bb541f611c7a7a5629e70a4e09aa8a4/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/cb6a2d88f01c0ae71314712b0fa3f19af6a35d2d7cb85eee42a9c4b0de317603/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/63bf9feb794cb49a5bfa185da9cfd0098de4334162aba7aa2a7fdc97331dbb20/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/0dac919a44c360d3e5ed14298dc364fa9506d4c585f6ff4d240e8a3b3a89954f/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/cf220cf1b68a9cf7d5b04374995bda5bab9ea5c68fbc22952d2e516ee751986c/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/73b8301526d7b8c8bf32242c5e0fd366e3aa31084bdb93d24c5336fe7714a5b7/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/70ef96f65e456b92ea29d0d16640a84c74a6d9d3d7e697e67ef7ea3c89955628/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/3c2499f44dfef226cf381cf14ba4be4195f26a8555eb428e0f41be24ceb71694/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/67ba865022681ae27d16d8a33759bbc7f8e606e2db030370b6ca359712da7c53/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/18de0fe0d84477462d9cb44594dd2f606125b8477516711a4e51b011826dce2c/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/1c2b67941e62fcba731cfa5db883b78f79f92226b5db971e18f45d29cc463d18/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/2bd68b7f65b023fb239f93a7d2e3bf46c0b59ea36b4986f14f9d1420f2dc2927/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/496c73238db64bd78d863191dd8e14b5ad0f17c9f4f8905d14e66b6940655063/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/7c7e8c97c15d14a92f5bb81e7387b9581fd47acc04cc72631e39470dca28fced/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/713603f0e5f4437eeb3bf35d5e0b25b82445e0b7be747df501b27c5f3d7c4fd2/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/5db684336aa62807042ba41117f53d03ac62d1d35b4dd92004a569ff81b3080f/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 02 22:52:31.094561 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.093643 5076 manager.go:217] Machine: {Timestamp:2026-11-02 22:52:31.09317762 +0000 UTC m=+0.061709394 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099792 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec227d0787f60e8b090322d0aa81776b SystemUUID:ec227d07-87f6-0e8b-0903-22d0aa81776b BootID:5a4a1bc2-bce4-4bfd-961e-8ec63843ac65 Filesystems:[{Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/46eae5da-cf0b-4663-bdeb-f8220fa08c05/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ef782737103253b57fdf6f5c12cb642b00ca83215d5a52e554ce2f17e691f49/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/914d69658f5c63ff96087f02576c562a6e1167cc1722fef6a93a4c62118ab3ab/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b1a2da6bba00ba8c0594bb3dbbeb3cca05136ff2fdbe870e87d5d1cbbfa6254a/userdata/shm DeviceMajor:0 DeviceMinor:245 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/df418d6f-b9d3-4019-967c-9511be123e1f/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/696172ee14d01e9d35bdac289d3290e2adc725e4cafa322f60dc04e3cb208c47/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3a464bf3-77fa-4fcd-8593-327fa1f4a19a/volumes/kubernetes.io~projected/kube-api-access-cml47 DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fd96552a-1a78-4fac-8faf-71d36f010905/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:286 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/25637d3d0737f9e52181634fdb879f8dde56f0b2436d7b7e695a500d72cab169/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/25ce3b94c1b208749a7a75bead39555de3944bf63e6d8f360ca695541ce8f157/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/75f7d1d7-88e4-41e0-bdd3-a43df363b408/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/de924197b3afd07e8c6b2d9ab15d26217b20b69c974fcc935146dd25598040e8/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cb1743a1-6e7b-4f99-80e6-197a6916580d/volumes/kubernetes.io~projected/kube-api-access-6rt8s DeviceMajor:0 DeviceMinor:51 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/0237eeb1-8c85-475e-8580-6dec324de705/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0237eeb1-8c85-475e-8580-6dec324de705/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/46eae5da-cf0b-4663-bdeb-f8220fa08c05/volumes/kubernetes.io~projected/kube-api-access-nkzvb DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/318c3a34f01581248a349ccfe0a15df08396d1a44a40b4c33f781061059a387e/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:49 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ac4a7fda-863e-4fb1-a98c-9b325a8c0df7/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e3a3bc57-6813-441c-bdc0-d39eb8214c61/volumes/kubernetes.io~projected/kube-api-access-q7nrp DeviceMajor:0 DeviceMinor:287 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d12c211b-f607-43b9-b621-12f061f109d9/volumes/kubernetes.io~projected/kube-api-access-bx9mw DeviceMajor:0 DeviceMinor:336 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5d546ba5bc54172ecbea70ee0fbfebe48033ebf77bc7a94c95663240295e4292/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/54ad588bb8e86ed38f7a08e74739b501f3663a3a0f5cda683afea2c00d975db2/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/58436ee67e99f72594ebf75d78e03a97fbc28924da7c08d072582970e23d689e/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d008a27-d5ba-49cb-b3fb-abdecd723abd/volumes/kubernetes.io~projected/kube-api-access-fqdk5 DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fbcab4d0-9dd6-4682-be01-c4f589697e95/volumes/kubernetes.io~projected/kube-api-access-24smc DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/236450f1-d9fa-488b-9022-ab4f5517549f/volumes/kubernetes.io~projected/kube-api-access-vxxqc DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/924af956a0c2c62f133a420c317fb5dd6c2d5bbfde22749bfceb02dc429d5b67/userdata/shm DeviceMajor:0 DeviceMinor:330 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a7eef4c62c90023e339d07ef34aa6a936384058d68431afa17a67efc0579b03e/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eb0cdddb2bd55368782469edc79d2349ea253148a8e8af8a37591b6f4181ba61/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/105406c8-6800-4562-9651-962bee9f8d73/volumes/kubernetes.io~projected/kube-api-access-9f4r4 DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/df418d6f-b9d3-4019-967c-9511be123e1f/volumes/kubernetes.io~projected/kube-api-access-jjljw DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/46eae5da-cf0b-4663-bdeb-f8220fa08c05/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/42d598be-a6f1-486e-9ac3-2f5ece575f89/volumes/kubernetes.io~projected/kube-api-access-stv25 DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0237eeb1-8c85-475e-8580-6dec324de705/volumes/kubernetes.io~projected/kube-api-access-nwnxj DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/87ead83b47d5f3fc7380e32329e4f44fd3f61c3f216506076b57c9c75269f0d5/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f38c14908ece618cc8f271a7b83d2be46b29f52d2bcf1ca5db920bf6436246f8/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/50d13ae8-4ab5-4ca7-a11a-ad632da19669/volumes/kubernetes.io~projected/kube-api-access-5j4hf DeviceMajor:0 DeviceMinor:327 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/387a216b6d58d9b749b3b5ca2cc899e1729c8ed618fd1fd4517f1cc35170c68e/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/31ea80300d42577ad470fb6f4391ef10c997d0836355f54625ad14cb535f4483/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/00b95c29-7df8-4602-bd0b-06b6281d5dc1/volumes/kubernetes.io~projected/kube-api-access-zt847 DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/75f7d1d7-88e4-41e0-bdd3-a43df363b408/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4b9cd8c7-228b-451a-afe4-3af7a3bf0a42/volumes/kubernetes.io~projected/kube-api-access-ncmnm DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b6b45c3-5275-4877-a84b-15e602100c40/volumes/kubernetes.io~projected/kube-api-access-6flqc DeviceMajor:0 DeviceMinor:337 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9332cac3ad414a7a953cd8f44d53bed498e0df65d2ec5b79aae1ba771af997e6/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6b3753c7-7634-4209-8760-7a52da534549/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:50 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fc69f07f-4001-4951-aff8-d3d6671cc5d4/volumes/kubernetes.io~projected/kube-api-access-n6jlv DeviceMajor:0 DeviceMinor:53 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b858466f-3dcd-419c-9495-4dbb89e8d51b/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/60832943-bf1f-4685-be00-08152c423b08/volumes/kubernetes.io~projected/kube-api-access-pmk5v DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/75f7d1d7-88e4-41e0-bdd3-a43df363b408/volumes/kubernetes.io~projected/kube-api-access-md7ml DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d3575a78-aad1-41e0-bad4-63724365aeb1/volumes/kubernetes.io~projected/kube-api-access-vvg2k DeviceMajor:0 DeviceMinor:54 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4f48f38b5e24171bd945b75111ee6fa5347a41c456770b64e8f7d43d1ee65e38/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/volumes/kubernetes.io~projected/kube-api-access-89zw7 DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/15789a0d-e030-45e0-a129-41de738ef767/volumes/kubernetes.io~projected/kube-api-access-jnt9q DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/79e2a74e-667b-4851-9e8a-759cb2a91f98/volumes/kubernetes.io~projected/kube-api-access-lx46q DeviceMajor:0 DeviceMinor:361 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8fbcc658-b377-4da4-bf1d-e6274dfd805b/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:48 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6b3753c7-7634-4209-8760-7a52da534549/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:47 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ee52c167-6361-4adb-9899-93ca4c2128b8/volumes/kubernetes.io~projected/kube-api-access-q7qr5 DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/055a42e181d047b2d0d0a3eb5266a6afecee615a52a63186bcdfc22ce49167c3/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/79e2a74e-667b-4851-9e8a-759cb2a91f98/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:354 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ee0f5d8810757aa96638f8b91b9e18c1185af1ff8cb70d801bfedb7b95ddda50/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/923bdca363bdecec9681204d8c7d61a686ad35aad15b349b72375be1ca7ac71e/userdata/shm DeviceMajor:0 DeviceMinor:338 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/668094b0-0e10-47aa-ba49-6944f7083bc6/volumes/kubernetes.io~projected/kube-api-access-k4shq DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ac4a7fda-863e-4fb1-a98c-9b325a8c0df7/volumes/kubernetes.io~projected/kube-api-access-9t8ms DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6b3753c7-7634-4209-8760-7a52da534549/volumes/kubernetes.io~projected/kube-api-access-wzxgm DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9f98166f-8051-4dbd-9856-db692932050b/volumes/kubernetes.io~projected/kube-api-access-85k6p DeviceMajor:0 DeviceMinor:322 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49/volumes/kubernetes.io~projected/kube-api-access-kkqkc DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/47f55c937c827013ee62eecf35fb78175c77ee2729e44dbf3f8d37893e51eb7d/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4f8b41ab-cbec-4f72-a5d9-853953748c0b/volumes/kubernetes.io~projected/kube-api-access-ctsxp DeviceMajor:0 DeviceMinor:52 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bae187d60c71e81ac2558a9049659dfeb1817e080bfab2d74e4c2e79695056ee/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/edff1bb6209a6e5782a55a8157d2c1884fef8ac84da3480932b04f9c9ebebcec/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/60832943-bf1f-4685-be00-08152c423b08/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/46eae5da-cf0b-4663-bdeb-f8220fa08c05/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b858466f-3dcd-419c-9495-4dbb89e8d51b/volumes/kubernetes.io~projected/kube-api-access-ws5j7 DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0237eeb1-8c85-475e-8580-6dec324de705/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:055a42e181d047b MacAddress:8e:78:df:b9:82:b5 Speed:10000 Mtu:8901} {Name:318c3a34f015812 MacAddress:12:b1:4f:2f:34:b6 Speed:10000 Mtu:8901} {Name:31ea80300d42577 MacAddress:da:da:ce:35:c7:65 Speed:10000 Mtu:8901} {Name:387a216b6d58d9b MacAddress:12:6e:25:60:0a:ae Speed:10000 Mtu:8901} {Name:47f55c937c82701 MacAddress:7a:39:54:d0:41:22 Speed:10000 Mtu:8901} {Name:54ad588bb8e86ed MacAddress:0a:4a:a1:f0:f5:23 Speed:10000 Mtu:8901} {Name:696172ee14d01e9 MacAddress:fa:60:f4:87:ef:ad Speed:10000 Mtu:8901} {Name:87ead83b47d5f3f MacAddress:72:5d:d1:1f:a7:bf Speed:10000 Mtu:8901} {Name:923bdca363bdece MacAddress:12:66:31:58:51:ab Speed:10000 Mtu:8901} {Name:924af956a0c2c62 MacAddress:a2:77:06:20:43:5d Speed:10000 Mtu:8901} {Name:9332cac3ad414a7 MacAddress:32:99:62:94:2f:ec Speed:10000 Mtu:8901} {Name:b1a2da6bba00ba8 MacAddress:76:ab:2b:f0:b2:b0 Speed:10000 Mtu:8901} {Name:bae187d60c71e81 MacAddress:9e:6d:7f:ff:a4:ba Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:e9:c6:37:91 Speed:0 Mtu:9001} {Name:br-int MacAddress:4a:b6:6d:f2:18:57 Speed:0 Mtu:8901} {Name:de924197b3afd07 MacAddress:56:a2:cb:9f:29:c1 Speed:10000 Mtu:8901} {Name:ee0f5d8810757aa MacAddress:06:ae:4a:05:0b:13 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:e9:c6:37:91 Speed:0 Mtu:9001} {Name:f38c14908ece618 MacAddress:ee:34:7d:e0:91:7c Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:6e:9a:8c:73:7d:33 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ca:1b:6d:74:97:91 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 22:52:31.094561 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.094022 5076 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 22:52:31.094561 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.094069 5076 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 22:52:31.094561 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.094356 5076 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 22:52:31.094561 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.094377 5076 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-211.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 22:52:31.094561 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.094522 5076 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 22:52:31.094561 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.094530 5076 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 22:52:31.094561 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.094545 5076 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 22:52:31.094561 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.094565 5076 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 22:52:31.094972 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.094585 5076 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 22:52:31.094972 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.094742 5076 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 22:52:31.094972 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.094851 5076 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 22:52:31.094972 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.094901 5076 kubelet.go:493] "Attempting to sync node with API server" Nov 02 22:52:31.094972 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.094911 5076 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 22:52:31.094972 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.094925 5076 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 22:52:31.094972 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.094934 5076 kubelet.go:405] "Adding apiserver pod source" Nov 02 22:52:31.094972 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.094943 5076 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 22:52:31.095652 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.095632 5076 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 22:52:31.095725 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.095658 5076 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 22:52:31.096155 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.096125 5076 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 22:52:31.099558 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.099541 5076 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 22:52:31.099558 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.099555 5076 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 22:52:31.099667 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.099649 5076 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 22:52:31.099844 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.099833 5076 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 22:52:31.099875 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.099862 5076 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 22:52:31.099979 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.099970 5076 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 22:52:31.100016 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.099985 5076 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 22:52:31.100016 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.099991 5076 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 22:52:31.100016 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.099997 5076 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 22:52:31.100016 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.100003 5076 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 22:52:31.100016 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.100009 5076 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 22:52:31.100016 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.100015 5076 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 22:52:31.100177 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.100020 5076 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 22:52:31.100177 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.100027 5076 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 22:52:31.100177 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.100033 5076 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 22:52:31.100177 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.100042 5076 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 22:52:31.100177 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.100051 5076 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 22:52:31.100177 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.100075 5076 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 22:52:31.100177 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.100081 5076 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 22:52:31.103572 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.103558 5076 server.go:1263] "Started kubelet" Nov 02 22:52:31.103666 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.103632 5076 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 22:52:31.103719 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.103650 5076 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 22:52:31.103766 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.103726 5076 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 22:52:31.104089 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.104070 5076 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 22:52:31.104148 ip-10-0-130-211 systemd[1]: Started Kubernetes Kubelet. Nov 02 22:52:31.104962 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.104944 5076 server.go:317] "Adding debug handlers to kubelet server" Nov 02 22:52:31.109635 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.107156 5076 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 22:52:31.109635 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.109228 5076 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 22:52:31.110741 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:31.110724 5076 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-211.us-west-2.compute.internal\" not found" Nov 02 22:52:31.110868 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.110856 5076 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 22:52:31.110978 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.110969 5076 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 22:52:31.111251 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.111229 5076 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 22:52:31.111476 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.111425 5076 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 22:46:49 +0000 UTC" deadline="2028-06-09 02:06:55.176553743 +0000 UTC" Nov 02 22:52:31.111476 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.111473 5076 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14019h14m24.065084013s" Nov 02 22:52:31.111871 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.111708 5076 factory.go:55] Registering systemd factory Nov 02 22:52:31.111871 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.111733 5076 factory.go:223] Registration of the systemd container factory successfully Nov 02 22:52:31.113140 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.113119 5076 factory.go:153] Registering CRI-O factory Nov 02 22:52:31.113247 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.113238 5076 factory.go:223] Registration of the crio container factory successfully Nov 02 22:52:31.113358 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.113349 5076 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 22:52:31.113451 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.113442 5076 factory.go:103] Registering Raw factory Nov 02 22:52:31.113520 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.113512 5076 manager.go:1199] Started watching for new ooms in manager Nov 02 22:52:31.113947 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.113935 5076 manager.go:319] Starting recovery of all containers Nov 02 22:52:31.115861 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.115811 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b858466f-3dcd-419c-9495-4dbb89e8d51b" volumeName="kubernetes.io/configmap/b858466f-3dcd-419c-9495-4dbb89e8d51b-service-ca-bundle" seLinuxMountContext="" Nov 02 22:52:31.115969 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.115877 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b858466f-3dcd-419c-9495-4dbb89e8d51b" volumeName="kubernetes.io/configmap/b858466f-3dcd-419c-9495-4dbb89e8d51b-trusted-ca-bundle" seLinuxMountContext="" Nov 02 22:52:31.115969 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.115909 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="105406c8-6800-4562-9651-962bee9f8d73" volumeName="kubernetes.io/configmap/105406c8-6800-4562-9651-962bee9f8d73-config-volume" seLinuxMountContext="" Nov 02 22:52:31.115969 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.115932 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49" volumeName="kubernetes.io/configmap/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-env-overrides" seLinuxMountContext="" Nov 02 22:52:31.115969 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.115947 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46eae5da-cf0b-4663-bdeb-f8220fa08c05" volumeName="kubernetes.io/empty-dir/46eae5da-cf0b-4663-bdeb-f8220fa08c05-ca-trust-extracted" seLinuxMountContext="" Nov 02 22:52:31.116169 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.115962 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b9cd8c7-228b-451a-afe4-3af7a3bf0a42" volumeName="kubernetes.io/projected/4b9cd8c7-228b-451a-afe4-3af7a3bf0a42-kube-api-access-ncmnm" seLinuxMountContext="" Nov 02 22:52:31.116169 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.115991 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="668094b0-0e10-47aa-ba49-6944f7083bc6" volumeName="kubernetes.io/configmap/668094b0-0e10-47aa-ba49-6944f7083bc6-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 22:52:31.116169 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116004 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b3753c7-7634-4209-8760-7a52da534549" volumeName="kubernetes.io/projected/6b3753c7-7634-4209-8760-7a52da534549-kube-api-access-wzxgm" seLinuxMountContext="" Nov 02 22:52:31.116169 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116025 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79e2a74e-667b-4851-9e8a-759cb2a91f98" volumeName="kubernetes.io/configmap/79e2a74e-667b-4851-9e8a-759cb2a91f98-signing-cabundle" seLinuxMountContext="" Nov 02 22:52:31.116169 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116040 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b858466f-3dcd-419c-9495-4dbb89e8d51b" volumeName="kubernetes.io/projected/b858466f-3dcd-419c-9495-4dbb89e8d51b-kube-api-access-ws5j7" seLinuxMountContext="" Nov 02 22:52:31.116169 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116057 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0237eeb1-8c85-475e-8580-6dec324de705" volumeName="kubernetes.io/configmap/0237eeb1-8c85-475e-8580-6dec324de705-trusted-ca" seLinuxMountContext="" Nov 02 22:52:31.116169 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116071 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0237eeb1-8c85-475e-8580-6dec324de705" volumeName="kubernetes.io/secret/0237eeb1-8c85-475e-8580-6dec324de705-installation-pull-secrets" seLinuxMountContext="" Nov 02 22:52:31.116169 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116084 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60832943-bf1f-4685-be00-08152c423b08" volumeName="kubernetes.io/secret/60832943-bf1f-4685-be00-08152c423b08-serving-cert" seLinuxMountContext="" Nov 02 22:52:31.116169 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116102 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75f7d1d7-88e4-41e0-bdd3-a43df363b408" volumeName="kubernetes.io/projected/75f7d1d7-88e4-41e0-bdd3-a43df363b408-kube-api-access-md7ml" seLinuxMountContext="" Nov 02 22:52:31.116169 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116120 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb1743a1-6e7b-4f99-80e6-197a6916580d" volumeName="kubernetes.io/projected/cb1743a1-6e7b-4f99-80e6-197a6916580d-kube-api-access-6rt8s" seLinuxMountContext="" Nov 02 22:52:31.116169 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116138 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d12c211b-f607-43b9-b621-12f061f109d9" volumeName="kubernetes.io/configmap/d12c211b-f607-43b9-b621-12f061f109d9-service-ca-bundle" seLinuxMountContext="" Nov 02 22:52:31.116169 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116150 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee52c167-6361-4adb-9899-93ca4c2128b8" volumeName="kubernetes.io/projected/ee52c167-6361-4adb-9899-93ca4c2128b8-kube-api-access-q7qr5" seLinuxMountContext="" Nov 02 22:52:31.116169 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116167 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00b95c29-7df8-4602-bd0b-06b6281d5dc1" volumeName="kubernetes.io/projected/00b95c29-7df8-4602-bd0b-06b6281d5dc1-kube-api-access-zt847" seLinuxMountContext="" Nov 02 22:52:31.116686 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116179 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="105406c8-6800-4562-9651-962bee9f8d73" volumeName="kubernetes.io/projected/105406c8-6800-4562-9651-962bee9f8d73-kube-api-access-9f4r4" seLinuxMountContext="" Nov 02 22:52:31.116686 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116199 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a464bf3-77fa-4fcd-8593-327fa1f4a19a" volumeName="kubernetes.io/projected/3a464bf3-77fa-4fcd-8593-327fa1f4a19a-kube-api-access-cml47" seLinuxMountContext="" Nov 02 22:52:31.116686 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116211 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75f7d1d7-88e4-41e0-bdd3-a43df363b408" volumeName="kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-default-certificate" seLinuxMountContext="" Nov 02 22:52:31.116686 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116224 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b858466f-3dcd-419c-9495-4dbb89e8d51b" volumeName="kubernetes.io/secret/b858466f-3dcd-419c-9495-4dbb89e8d51b-serving-cert" seLinuxMountContext="" Nov 02 22:52:31.116686 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116242 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbcab4d0-9dd6-4682-be01-c4f589697e95" volumeName="kubernetes.io/projected/fbcab4d0-9dd6-4682-be01-c4f589697e95-kube-api-access-24smc" seLinuxMountContext="" Nov 02 22:52:31.116686 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116255 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00b95c29-7df8-4602-bd0b-06b6281d5dc1" volumeName="kubernetes.io/configmap/00b95c29-7df8-4602-bd0b-06b6281d5dc1-samples-operator-config" seLinuxMountContext="" Nov 02 22:52:31.116686 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116271 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0237eeb1-8c85-475e-8580-6dec324de705" volumeName="kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-kube-api-access-nwnxj" seLinuxMountContext="" Nov 02 22:52:31.116686 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116284 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15789a0d-e030-45e0-a129-41de738ef767" volumeName="kubernetes.io/configmap/15789a0d-e030-45e0-a129-41de738ef767-telemetry-config" seLinuxMountContext="" Nov 02 22:52:31.116686 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116296 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b6b45c3-5275-4877-a84b-15e602100c40" volumeName="kubernetes.io/projected/0b6b45c3-5275-4877-a84b-15e602100c40-kube-api-access-6flqc" seLinuxMountContext="" Nov 02 22:52:31.116686 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116314 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49" volumeName="kubernetes.io/configmap/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-ovnkube-script-lib" seLinuxMountContext="" Nov 02 22:52:31.116686 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116339 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46eae5da-cf0b-4663-bdeb-f8220fa08c05" volumeName="kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-bound-sa-token" seLinuxMountContext="" Nov 02 22:52:31.116686 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116352 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f8b41ab-cbec-4f72-a5d9-853953748c0b" volumeName="kubernetes.io/configmap/4f8b41ab-cbec-4f72-a5d9-853953748c0b-iptables-alerter-script" seLinuxMountContext="" Nov 02 22:52:31.116686 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116364 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fbcc658-b377-4da4-bf1d-e6274dfd805b" volumeName="kubernetes.io/secret/8fbcc658-b377-4da4-bf1d-e6274dfd805b-agent-certs" seLinuxMountContext="" Nov 02 22:52:31.116686 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116381 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac4a7fda-863e-4fb1-a98c-9b325a8c0df7" volumeName="kubernetes.io/secret/ac4a7fda-863e-4fb1-a98c-9b325a8c0df7-cert" seLinuxMountContext="" Nov 02 22:52:31.116686 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116393 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3575a78-aad1-41e0-bad4-63724365aeb1" volumeName="kubernetes.io/empty-dir/d3575a78-aad1-41e0-bad4-63724365aeb1-tmp-dir" seLinuxMountContext="" Nov 02 22:52:31.116686 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116412 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3575a78-aad1-41e0-bad4-63724365aeb1" volumeName="kubernetes.io/projected/d3575a78-aad1-41e0-bad4-63724365aeb1-kube-api-access-vvg2k" seLinuxMountContext="" Nov 02 22:52:31.116686 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116425 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f8b41ab-cbec-4f72-a5d9-853953748c0b" volumeName="kubernetes.io/projected/4f8b41ab-cbec-4f72-a5d9-853953748c0b-kube-api-access-ctsxp" seLinuxMountContext="" Nov 02 22:52:31.116686 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116439 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6c7074b-4caa-43fb-9fca-14ebb5abf958" volumeName="kubernetes.io/configmap/b6c7074b-4caa-43fb-9fca-14ebb5abf958-nginx-conf" seLinuxMountContext="" Nov 02 22:52:31.116686 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116456 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df418d6f-b9d3-4019-967c-9511be123e1f" volumeName="kubernetes.io/configmap/df418d6f-b9d3-4019-967c-9511be123e1f-config" seLinuxMountContext="" Nov 02 22:52:31.116686 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116469 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3a3bc57-6813-441c-bdc0-d39eb8214c61" volumeName="kubernetes.io/projected/e3a3bc57-6813-441c-bdc0-d39eb8214c61-kube-api-access-q7nrp" seLinuxMountContext="" Nov 02 22:52:31.116686 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116487 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc69f07f-4001-4951-aff8-d3d6671cc5d4" volumeName="kubernetes.io/projected/fc69f07f-4001-4951-aff8-d3d6671cc5d4-kube-api-access-n6jlv" seLinuxMountContext="" Nov 02 22:52:31.116686 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116500 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46eae5da-cf0b-4663-bdeb-f8220fa08c05" volumeName="kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-kube-api-access-nkzvb" seLinuxMountContext="" Nov 02 22:52:31.116686 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116513 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60832943-bf1f-4685-be00-08152c423b08" volumeName="kubernetes.io/configmap/60832943-bf1f-4685-be00-08152c423b08-config" seLinuxMountContext="" Nov 02 22:52:31.116686 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116532 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="668094b0-0e10-47aa-ba49-6944f7083bc6" volumeName="kubernetes.io/projected/668094b0-0e10-47aa-ba49-6944f7083bc6-kube-api-access-k4shq" seLinuxMountContext="" Nov 02 22:52:31.116686 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116544 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b858466f-3dcd-419c-9495-4dbb89e8d51b" volumeName="kubernetes.io/empty-dir/b858466f-3dcd-419c-9495-4dbb89e8d51b-snapshots" seLinuxMountContext="" Nov 02 22:52:31.116686 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116560 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df418d6f-b9d3-4019-967c-9511be123e1f" volumeName="kubernetes.io/projected/df418d6f-b9d3-4019-967c-9511be123e1f-kube-api-access-jjljw" seLinuxMountContext="" Nov 02 22:52:31.116686 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116574 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0237eeb1-8c85-475e-8580-6dec324de705" volumeName="kubernetes.io/empty-dir/0237eeb1-8c85-475e-8580-6dec324de705-ca-trust-extracted" seLinuxMountContext="" Nov 02 22:52:31.116686 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116587 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46eae5da-cf0b-4663-bdeb-f8220fa08c05" volumeName="kubernetes.io/configmap/46eae5da-cf0b-4663-bdeb-f8220fa08c05-registry-certificates" seLinuxMountContext="" Nov 02 22:52:31.116686 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116603 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46eae5da-cf0b-4663-bdeb-f8220fa08c05" volumeName="kubernetes.io/secret/46eae5da-cf0b-4663-bdeb-f8220fa08c05-installation-pull-secrets" seLinuxMountContext="" Nov 02 22:52:31.116686 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116615 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60832943-bf1f-4685-be00-08152c423b08" volumeName="kubernetes.io/projected/60832943-bf1f-4685-be00-08152c423b08-kube-api-access-pmk5v" seLinuxMountContext="" Nov 02 22:52:31.116686 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116631 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c4b6fbd-74cf-41c6-bbae-92748ac25e1e" volumeName="kubernetes.io/projected/9c4b6fbd-74cf-41c6-bbae-92748ac25e1e-kube-api-access-89zw7" seLinuxMountContext="" Nov 02 22:52:31.116686 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116643 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b858466f-3dcd-419c-9495-4dbb89e8d51b" volumeName="kubernetes.io/empty-dir/b858466f-3dcd-419c-9495-4dbb89e8d51b-tmp" seLinuxMountContext="" Nov 02 22:52:31.116686 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116655 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d12c211b-f607-43b9-b621-12f061f109d9" volumeName="kubernetes.io/empty-dir/d12c211b-f607-43b9-b621-12f061f109d9-archives-path" seLinuxMountContext="" Nov 02 22:52:31.116686 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116675 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0237eeb1-8c85-475e-8580-6dec324de705" volumeName="kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-bound-sa-token" seLinuxMountContext="" Nov 02 22:52:31.116686 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116688 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d008a27-d5ba-49cb-b3fb-abdecd723abd" volumeName="kubernetes.io/projected/0d008a27-d5ba-49cb-b3fb-abdecd723abd-kube-api-access-fqdk5" seLinuxMountContext="" Nov 02 22:52:31.118156 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116705 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79e2a74e-667b-4851-9e8a-759cb2a91f98" volumeName="kubernetes.io/projected/79e2a74e-667b-4851-9e8a-759cb2a91f98-kube-api-access-lx46q" seLinuxMountContext="" Nov 02 22:52:31.118156 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116718 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac4a7fda-863e-4fb1-a98c-9b325a8c0df7" volumeName="kubernetes.io/projected/ac4a7fda-863e-4fb1-a98c-9b325a8c0df7-kube-api-access-9t8ms" seLinuxMountContext="" Nov 02 22:52:31.118156 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116730 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd96552a-1a78-4fac-8faf-71d36f010905" volumeName="kubernetes.io/secret/fd96552a-1a78-4fac-8faf-71d36f010905-original-pull-secret" seLinuxMountContext="" Nov 02 22:52:31.118156 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116764 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15789a0d-e030-45e0-a129-41de738ef767" volumeName="kubernetes.io/projected/15789a0d-e030-45e0-a129-41de738ef767-kube-api-access-jnt9q" seLinuxMountContext="" Nov 02 22:52:31.118156 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116778 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79e2a74e-667b-4851-9e8a-759cb2a91f98" volumeName="kubernetes.io/secret/79e2a74e-667b-4851-9e8a-759cb2a91f98-signing-key" seLinuxMountContext="" Nov 02 22:52:31.118156 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116794 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c4b6fbd-74cf-41c6-bbae-92748ac25e1e" volumeName="kubernetes.io/secret/9c4b6fbd-74cf-41c6-bbae-92748ac25e1e-serving-cert" seLinuxMountContext="" Nov 02 22:52:31.118156 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116806 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49" volumeName="kubernetes.io/projected/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-kube-api-access-kkqkc" seLinuxMountContext="" Nov 02 22:52:31.118156 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116819 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49" volumeName="kubernetes.io/secret/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 22:52:31.118156 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116836 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b3753c7-7634-4209-8760-7a52da534549" volumeName="kubernetes.io/empty-dir/6b3753c7-7634-4209-8760-7a52da534549-tmp" seLinuxMountContext="" Nov 02 22:52:31.118156 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116848 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c4b6fbd-74cf-41c6-bbae-92748ac25e1e" volumeName="kubernetes.io/configmap/9c4b6fbd-74cf-41c6-bbae-92748ac25e1e-config" seLinuxMountContext="" Nov 02 22:52:31.118156 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116865 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df418d6f-b9d3-4019-967c-9511be123e1f" volumeName="kubernetes.io/secret/df418d6f-b9d3-4019-967c-9511be123e1f-serving-cert" seLinuxMountContext="" Nov 02 22:52:31.118156 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116877 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0237eeb1-8c85-475e-8580-6dec324de705" volumeName="kubernetes.io/configmap/0237eeb1-8c85-475e-8580-6dec324de705-registry-certificates" seLinuxMountContext="" Nov 02 22:52:31.118156 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116904 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="668094b0-0e10-47aa-ba49-6944f7083bc6" volumeName="kubernetes.io/configmap/668094b0-0e10-47aa-ba49-6944f7083bc6-cni-binary-copy" seLinuxMountContext="" Nov 02 22:52:31.118156 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116921 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b3753c7-7634-4209-8760-7a52da534549" volumeName="kubernetes.io/empty-dir/6b3753c7-7634-4209-8760-7a52da534549-etc-tuned" seLinuxMountContext="" Nov 02 22:52:31.118156 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116935 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c4b6fbd-74cf-41c6-bbae-92748ac25e1e" volumeName="kubernetes.io/configmap/9c4b6fbd-74cf-41c6-bbae-92748ac25e1e-trusted-ca" seLinuxMountContext="" Nov 02 22:52:31.118156 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116952 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f98166f-8051-4dbd-9856-db692932050b" volumeName="kubernetes.io/empty-dir/9f98166f-8051-4dbd-9856-db692932050b-data-volume" seLinuxMountContext="" Nov 02 22:52:31.118156 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116973 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee52c167-6361-4adb-9899-93ca4c2128b8" volumeName="kubernetes.io/configmap/ee52c167-6361-4adb-9899-93ca4c2128b8-multus-daemon-config" seLinuxMountContext="" Nov 02 22:52:31.118156 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.116985 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="236450f1-d9fa-488b-9022-ab4f5517549f" volumeName="kubernetes.io/projected/236450f1-d9fa-488b-9022-ab4f5517549f-kube-api-access-vxxqc" seLinuxMountContext="" Nov 02 22:52:31.118156 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.117001 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46eae5da-cf0b-4663-bdeb-f8220fa08c05" volumeName="kubernetes.io/secret/46eae5da-cf0b-4663-bdeb-f8220fa08c05-image-registry-private-configuration" seLinuxMountContext="" Nov 02 22:52:31.118156 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.117013 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="668094b0-0e10-47aa-ba49-6944f7083bc6" volumeName="kubernetes.io/configmap/668094b0-0e10-47aa-ba49-6944f7083bc6-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 22:52:31.118156 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.117032 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f98166f-8051-4dbd-9856-db692932050b" volumeName="kubernetes.io/projected/9f98166f-8051-4dbd-9856-db692932050b-kube-api-access-85k6p" seLinuxMountContext="" Nov 02 22:52:31.118156 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.117043 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d12c211b-f607-43b9-b621-12f061f109d9" volumeName="kubernetes.io/configmap/d12c211b-f607-43b9-b621-12f061f109d9-trusted-ca-bundle" seLinuxMountContext="" Nov 02 22:52:31.118156 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.117056 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="105406c8-6800-4562-9651-962bee9f8d73" volumeName="kubernetes.io/empty-dir/105406c8-6800-4562-9651-962bee9f8d73-tmp-dir" seLinuxMountContext="" Nov 02 22:52:31.118156 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.117074 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49" volumeName="kubernetes.io/configmap/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-ovnkube-config" seLinuxMountContext="" Nov 02 22:52:31.118156 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.117087 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50d13ae8-4ab5-4ca7-a11a-ad632da19669" volumeName="kubernetes.io/projected/50d13ae8-4ab5-4ca7-a11a-ad632da19669-kube-api-access-5j4hf" seLinuxMountContext="" Nov 02 22:52:31.118156 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.117103 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75f7d1d7-88e4-41e0-bdd3-a43df363b408" volumeName="kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-stats-auth" seLinuxMountContext="" Nov 02 22:52:31.118156 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.117116 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fbcc658-b377-4da4-bf1d-e6274dfd805b" volumeName="kubernetes.io/configmap/8fbcc658-b377-4da4-bf1d-e6274dfd805b-konnectivity-ca" seLinuxMountContext="" Nov 02 22:52:31.118156 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.117129 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d12c211b-f607-43b9-b621-12f061f109d9" volumeName="kubernetes.io/projected/d12c211b-f607-43b9-b621-12f061f109d9-kube-api-access-bx9mw" seLinuxMountContext="" Nov 02 22:52:31.118156 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.117144 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46eae5da-cf0b-4663-bdeb-f8220fa08c05" volumeName="kubernetes.io/configmap/46eae5da-cf0b-4663-bdeb-f8220fa08c05-trusted-ca" seLinuxMountContext="" Nov 02 22:52:31.118156 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.117156 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee52c167-6361-4adb-9899-93ca4c2128b8" volumeName="kubernetes.io/configmap/ee52c167-6361-4adb-9899-93ca4c2128b8-cni-binary-copy" seLinuxMountContext="" Nov 02 22:52:31.118156 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.117207 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0237eeb1-8c85-475e-8580-6dec324de705" volumeName="kubernetes.io/secret/0237eeb1-8c85-475e-8580-6dec324de705-image-registry-private-configuration" seLinuxMountContext="" Nov 02 22:52:31.118156 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.117223 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a464bf3-77fa-4fcd-8593-327fa1f4a19a" volumeName="kubernetes.io/configmap/3a464bf3-77fa-4fcd-8593-327fa1f4a19a-serviceca" seLinuxMountContext="" Nov 02 22:52:31.118156 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.117235 5076 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42d598be-a6f1-486e-9ac3-2f5ece575f89" volumeName="kubernetes.io/projected/42d598be-a6f1-486e-9ac3-2f5ece575f89-kube-api-access-stv25" seLinuxMountContext="" Nov 02 22:52:31.118156 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.117251 5076 reconstruct.go:97] "Volume reconstruction finished" Nov 02 22:52:31.118156 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.117259 5076 reconciler.go:29] "Reconciler: start to sync state" Nov 02 22:52:31.137261 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.137232 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/1.log" Nov 02 22:52:31.138306 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.138266 5076 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 22:52:31.138834 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.138811 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/1.log" Nov 02 22:52:31.139854 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.139830 5076 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 22:52:31.139854 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.139856 5076 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 22:52:31.140020 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.139880 5076 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 22:52:31.140020 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:31.139942 5076 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 22:52:31.169596 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.169560 5076 generic.go:356] "Generic (PLEG): container finished" podID="f2cabf9747a94425892740afb80ec0fc" containerID="ca14beba5a668986c160950e31521f74eeb8b3baf0c8ac82db04fa420f1dc44b" exitCode=0 Nov 02 22:52:31.179593 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.179571 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/1.log" Nov 02 22:52:31.179706 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.179599 5076 generic.go:356] "Generic (PLEG): container finished" podID="9c4b6fbd-74cf-41c6-bbae-92748ac25e1e" containerID="61bc91e4752e543b447d5adcc7109202c5002dfd3c1c92a91e17d4830f0014e5" exitCode=255 Nov 02 22:52:31.192199 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.192161 5076 generic.go:356] "Generic (PLEG): container finished" podID="668094b0-0e10-47aa-ba49-6944f7083bc6" containerID="de97d17d30df0dc6968c21c9462490f96b52ce1d0365908428ab462c3f30063b" exitCode=0 Nov 02 22:52:31.192199 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.192198 5076 generic.go:356] "Generic (PLEG): container finished" podID="668094b0-0e10-47aa-ba49-6944f7083bc6" containerID="49b3f268de56c791d565646fb91148756abe602a63dfe4e49d4be1bd2d457cec" exitCode=0 Nov 02 22:52:31.192403 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.192209 5076 generic.go:356] "Generic (PLEG): container finished" podID="668094b0-0e10-47aa-ba49-6944f7083bc6" containerID="56c6e25b7361e6d92201ce0315b72296ea92eeb44bfd1c90b437f68292f4c716" exitCode=0 Nov 02 22:52:31.192403 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.192219 5076 generic.go:356] "Generic (PLEG): container finished" podID="668094b0-0e10-47aa-ba49-6944f7083bc6" containerID="656a1019ab4fada7b0d0b3842f59e447ba28bd247e528b012a90be4c49860907" exitCode=0 Nov 02 22:52:31.192403 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.192228 5076 generic.go:356] "Generic (PLEG): container finished" podID="668094b0-0e10-47aa-ba49-6944f7083bc6" containerID="8961f820b351f2772f7c0b9d7090d9e8aabd6904e744079229a03bef2f5fd88d" exitCode=0 Nov 02 22:52:31.192403 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.192272 5076 generic.go:356] "Generic (PLEG): container finished" podID="668094b0-0e10-47aa-ba49-6944f7083bc6" containerID="fecfdf9c325b12d132bba90fb5eba4090ea17eec3d79bd6e5d3e9406fcad0d9c" exitCode=0 Nov 02 22:52:31.210911 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:31.210867 5076 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-211.us-west-2.compute.internal\" not found" Nov 02 22:52:31.219922 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.219836 5076 manager.go:324] Recovery completed Nov 02 22:52:31.238277 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.238257 5076 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:52:31.239608 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.239587 5076 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-211.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:52:31.239720 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.239623 5076 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-211.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:52:31.239720 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.239633 5076 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-211.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:52:31.240346 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:31.240328 5076 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 02 22:52:31.241968 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.241952 5076 cpu_manager.go:226] "Starting" policy="none" Nov 02 22:52:31.241968 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.241963 5076 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:52:31.242075 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.241966 5076 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 22:52:31.242075 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.242039 5076 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 22:52:31.242184 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.242170 5076 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 22:52:31.242216 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.242185 5076 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 22:52:31.242216 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.242201 5076 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 22:52:31.242216 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.242206 5076 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 22:52:31.242216 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.242213 5076 policy_none.go:50] "Start" Nov 02 22:52:31.242326 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.242227 5076 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 22:52:31.242326 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.242237 5076 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 22:52:31.242376 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.242363 5076 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 22:52:31.242376 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.242372 5076 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 22:52:31.242430 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.242381 5076 policy_none.go:44] "Start" Nov 02 22:52:31.245315 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.245301 5076 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-211.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:52:31.245400 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.245327 5076 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-211.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:52:31.245400 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.245337 5076 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-211.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:52:31.247214 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.247181 5076 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 22:52:31.252701 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.252687 5076 manager.go:341] "Starting Device Plugin manager" Nov 02 22:52:31.252777 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:31.252713 5076 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 22:52:31.252777 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.252722 5076 server.go:87] "Starting device plugin registration server" Nov 02 22:52:31.253053 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.253041 5076 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 22:52:31.253087 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.253055 5076 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 22:52:31.253140 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.253123 5076 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 22:52:31.253572 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.253528 5076 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 22:52:31.253654 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.253582 5076 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 22:52:31.253654 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.253590 5076 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 22:52:31.254635 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.253986 5076 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T22:52:31.25356997Z","UUID":"15402aa2-e169-423c-8c9b-b6e5762eb3dd","Handler":null,"Name":"","Endpoint":""} Nov 02 22:52:31.256107 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:31.256079 5076 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 22:52:31.256300 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:31.256134 5076 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-211.us-west-2.compute.internal\" not found" Nov 02 22:52:31.256373 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.256360 5076 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 22:52:31.256414 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.256407 5076 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 22:52:31.293601 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.293573 5076 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:52:31.294695 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.294675 5076 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:52:31.295404 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.295179 5076 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 22:52:31.295404 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.295182 5076 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:52:31.342687 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.342647 5076 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:52:31.353731 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.353704 5076 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:52:31.354921 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.354903 5076 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-211.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:52:31.355037 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.354934 5076 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-211.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:52:31.355037 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.354946 5076 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-211.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:52:31.355037 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.354991 5076 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:52:31.440703 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.440624 5076 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-211.us-west-2.compute.internal"] Nov 02 22:52:31.441556 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.441450 5076 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:52:31.441556 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.441460 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal" event={"ID":"f2cabf9747a94425892740afb80ec0fc","Type":"ContainerStarted","Data":"5e34994a75d18bb5dec964ac44a1468f4db7795cbcd1467ada7e6e4fd1d58783"} Nov 02 22:52:31.441556 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.441517 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal" event={"ID":"f2cabf9747a94425892740afb80ec0fc","Type":"ContainerDied","Data":"ca14beba5a668986c160950e31521f74eeb8b3baf0c8ac82db04fa420f1dc44b"} Nov 02 22:52:31.441556 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.441542 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal" event={"ID":"f2cabf9747a94425892740afb80ec0fc","Type":"ContainerStarted","Data":"25637d3d0737f9e52181634fdb879f8dde56f0b2436d7b7e695a500d72cab169"} Nov 02 22:52:31.442128 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.441564 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-211.us-west-2.compute.internal" event={"ID":"7ba3fed7e8adcc2e1f404604f948f337","Type":"ContainerStarted","Data":"3c7e8a2b873bfc147557449a3285c21c8bb11eb9626f224d770ad62a61319e05"} Nov 02 22:52:31.442128 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.441582 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-211.us-west-2.compute.internal" event={"ID":"7ba3fed7e8adcc2e1f404604f948f337","Type":"ContainerStarted","Data":"edff1bb6209a6e5782a55a8157d2c1884fef8ac84da3480932b04f9c9ebebcec"} Nov 02 22:52:31.442128 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.441727 5076 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:52:31.459558 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.459533 5076 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:52:31.459669 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.459592 5076 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:52:31.500983 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.500919 5076 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:52:31.500983 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.500933 5076 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:52:31.500983 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:31.500959 5076 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-211.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:52:31.500983 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:31.500971 5076 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:52:31.596797 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.596759 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7ba3fed7e8adcc2e1f404604f948f337-config\") pod \"kube-apiserver-proxy-ip-10-0-130-211.us-west-2.compute.internal\" (UID: \"7ba3fed7e8adcc2e1f404604f948f337\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:52:31.596797 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.596800 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f2cabf9747a94425892740afb80ec0fc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal\" (UID: \"f2cabf9747a94425892740afb80ec0fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:52:31.596998 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.596830 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2cabf9747a94425892740afb80ec0fc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal\" (UID: \"f2cabf9747a94425892740afb80ec0fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:52:31.697978 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.697932 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2cabf9747a94425892740afb80ec0fc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal\" (UID: \"f2cabf9747a94425892740afb80ec0fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:52:31.698150 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.698041 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7ba3fed7e8adcc2e1f404604f948f337-config\") pod \"kube-apiserver-proxy-ip-10-0-130-211.us-west-2.compute.internal\" (UID: \"7ba3fed7e8adcc2e1f404604f948f337\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:52:31.698150 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.698116 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2cabf9747a94425892740afb80ec0fc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal\" (UID: \"f2cabf9747a94425892740afb80ec0fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:52:31.698150 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.697982 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7ba3fed7e8adcc2e1f404604f948f337-config\") pod \"kube-apiserver-proxy-ip-10-0-130-211.us-west-2.compute.internal\" (UID: \"7ba3fed7e8adcc2e1f404604f948f337\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:52:31.698297 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.698150 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f2cabf9747a94425892740afb80ec0fc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal\" (UID: \"f2cabf9747a94425892740afb80ec0fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:52:31.698297 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:31.698201 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f2cabf9747a94425892740afb80ec0fc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal\" (UID: \"f2cabf9747a94425892740afb80ec0fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:52:32.096020 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.095984 5076 apiserver.go:52] "Watching apiserver" Nov 02 22:52:32.203078 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.203047 5076 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:52:32.203240 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.203122 5076 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:52:32.243210 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.243172 5076 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 22:52:32.244065 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.244033 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal","openshift-multus/multus-njn94","openshift-console-operator/console-operator-7cd56b8dbc-c2gxc","openshift-monitoring/cluster-monitoring-operator-884576c57-mbmf9","openshift-image-registry/image-registry-7b5b8cd649-z4rbs","openshift-kube-storage-version-migrator/migrator-66df5d496b-8mxfd","openshift-kube-storage-version-migrator/migrator-66df5d496b-f62lt","openshift-network-diagnostics/network-check-source-6f78d59f99-6zkx2","openshift-insights/insights-runtime-extractor-sv7kh","openshift-ovn-kubernetes/ovnkube-node-bp6bf","kube-system/kube-apiserver-proxy-ip-10-0-130-211.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-9zqrz","openshift-network-console/networking-console-plugin-86c478bb88-kkzvx","kube-system/kas-connection-checker-685895c7c9-fdj8t","openshift-cluster-node-tuning-operator/tuned-zjn7n","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-867x9","kube-system/global-pull-secret-syncer-k4gfs","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv","openshift-ingress/router-default-896ccfb6c-q4rg5","openshift-network-diagnostics/network-check-target-rmgpt","openshift-network-operator/iptables-alerter-92hm7","openshift-service-ca/service-ca-5fffc97bdf-hrdqf","openshift-dns/node-resolver-nrxwb","openshift-insights/periodic-gathering-lx8nq-mmb4p","openshift-image-registry/image-registry-777f69544d-d75n6","openshift-insights/insights-operator-7c4986d449-xszmw","kube-system/kas-connection-checker-685895c7c9-mtrcx","kube-system/kas-connection-checker-685895c7c9-pxpxp","openshift-dns/dns-default-pvc5f","openshift-ingress-canary/ingress-canary-h9lck","kube-system/konnectivity-agent-zqssw","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ht25s","openshift-service-ca-operator/service-ca-operator-544d749c8c-p66wt","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cgpkk","openshift-multus/multus-additional-cni-plugins-cs2mc","openshift-image-registry/node-ca-4mhgx"] Nov 02 22:52:32.248008 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.247962 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:32.248191 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.248166 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbmf9" Nov 02 22:52:32.249108 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.249068 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cgpkk" Nov 02 22:52:32.251276 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.251257 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:32.251717 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.251701 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-kkzvx" Nov 02 22:52:32.253429 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.253411 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zqrz" Nov 02 22:52:32.253916 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.253902 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:32.259488 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.259470 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pvc5f" Nov 02 22:52:32.259960 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.259946 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sv7kh" Nov 02 22:52:32.260743 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.260727 5076 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:52:32.260815 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:32.260787 5076 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-211.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:52:32.261089 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.261069 5076 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:52:32.261177 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:32.261119 5076 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-211.us-west-2.compute.internal" Nov 02 22:52:32.294609 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.294577 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 02 22:52:32.297275 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.297250 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 02 22:52:32.299563 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.299541 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 02 22:52:32.299640 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.299563 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 22:52:32.299686 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.299642 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:52:32.299912 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.299880 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 02 22:52:32.299912 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.299884 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 02 22:52:32.299998 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.299923 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 22:52:32.299998 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.299933 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 02 22:52:32.299998 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.299943 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 02 22:52:32.299998 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.299964 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 02 22:52:32.300113 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.300004 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 02 22:52:32.300113 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.300011 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 22:52:32.300113 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.300014 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 02 22:52:32.300113 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.299900 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:52:32.300113 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.300077 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 22:52:32.300290 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.300187 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 02 22:52:32.300290 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.300215 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 02 22:52:32.300290 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.300254 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 02 22:52:32.300290 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.300270 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 02 22:52:32.300454 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.300328 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 02 22:52:32.300454 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.300372 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 22:52:32.300454 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.300439 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 22:52:32.300581 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.300457 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:52:32.300581 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.300481 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 02 22:52:32.300581 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.300483 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 02 22:52:32.301048 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.301027 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 02 22:52:32.301286 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.301267 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 02 22:52:32.301501 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.301477 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 02 22:52:32.301658 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.301643 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 02 22:52:32.301729 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.301711 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 22:52:32.301787 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.301774 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 02 22:52:32.302080 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.302056 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 02 22:52:32.302393 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.302328 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 22:52:32.302393 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.302393 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:52:32.302551 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.302537 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 02 22:52:32.308468 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.308449 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 02 22:52:32.308551 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.308516 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 22:52:32.313382 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.313368 5076 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 22:52:32.347780 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.347716 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 02 22:52:32.347923 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.347803 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:52:32.347923 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.347835 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 22:52:32.348585 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.348079 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 02 22:52:32.348585 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.348100 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 22:52:32.348585 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.348112 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 22:52:32.348585 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.348109 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 22:52:32.348585 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.348141 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 22:52:32.348585 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.348144 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 22:52:32.348585 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.348174 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 22:52:32.348585 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.348175 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 22:52:32.348585 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.348187 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 22:52:32.348585 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.348194 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 02 22:52:32.348585 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.348227 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 22:52:32.348585 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.348251 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 02 22:52:32.348585 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.348257 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 22:52:32.348585 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.348265 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 02 22:52:32.348585 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.348281 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 22:52:32.348585 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.348298 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 22:52:32.348585 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.348326 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:52:32.348585 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.348330 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 02 22:52:32.348585 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.348337 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 22:52:32.348585 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.348343 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 22:52:32.348585 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.348379 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 22:52:32.348585 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.348396 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 22:52:32.348585 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.348446 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 22:52:32.348585 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.348469 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 22:52:32.348585 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.348478 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 22:52:32.348585 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.348490 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 22:52:32.348585 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.348493 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:52:32.348585 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.348522 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 22:52:32.348585 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.348536 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 22:52:32.348585 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.348556 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 22:52:32.348585 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.348561 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 22:52:32.348585 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.348587 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 22:52:32.349589 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.348757 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 22:52:32.349589 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.349043 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 02 22:52:32.399655 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.399594 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 22:52:32.399818 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.399704 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vbflc\"" Nov 02 22:52:32.399818 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.399813 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 22:52:32.399964 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.399866 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 02 22:52:32.399964 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.399879 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-45bns\"" Nov 02 22:52:32.399964 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.399935 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2rjj4\"" Nov 02 22:52:32.399964 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.399937 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rxqrv\"" Nov 02 22:52:32.400146 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.399978 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 22:52:32.400146 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.399995 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-n2bfg\"" Nov 02 22:52:32.400146 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.400031 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 22:52:32.400146 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.400042 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9p98l\"" Nov 02 22:52:32.400146 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.400118 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 22:52:32.400354 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.400203 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-qg26z\"" Nov 02 22:52:32.400354 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.400209 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sdz5j\"" Nov 02 22:52:32.400354 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.400338 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-crtkq\"" Nov 02 22:52:32.400477 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.400353 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 22:52:32.400477 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.400384 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-92nqq\"" Nov 02 22:52:32.400477 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.400391 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 02 22:52:32.400477 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.400396 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5kxql\"" Nov 02 22:52:32.400477 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.400398 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-nxb2c\"" Nov 02 22:52:32.400477 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.400430 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t6gjv\"" Nov 02 22:52:32.400477 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.400463 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-r5gxd\"" Nov 02 22:52:32.400772 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.400517 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 02 22:52:32.400772 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.400528 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-8zgv4\"" Nov 02 22:52:32.400772 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.400595 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-c9lzr\"" Nov 02 22:52:32.400772 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.400655 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9qgnp\"" Nov 02 22:52:32.400772 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.400709 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-77brg\"" Nov 02 22:52:32.401008 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.400816 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 22:52:32.401008 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.400909 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 22:52:32.401008 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.400950 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-v5hvl\"" Nov 02 22:52:32.403915 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.403885 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/60832943-bf1f-4685-be00-08152c423b08-serving-cert\") pod \"service-ca-operator-544d749c8c-p66wt\" (UID: \"60832943-bf1f-4685-be00-08152c423b08\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p66wt" Nov 02 22:52:32.403982 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.403926 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0237eeb1-8c85-475e-8580-6dec324de705-image-registry-private-configuration\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:32.403982 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.403955 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/46eae5da-cf0b-4663-bdeb-f8220fa08c05-image-registry-private-configuration\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:32.404141 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.404123 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvg2k\" (UniqueName: \"kubernetes.io/projected/d3575a78-aad1-41e0-bad4-63724365aeb1-kube-api-access-vvg2k\") pod \"node-resolver-nrxwb\" (UID: \"d3575a78-aad1-41e0-bad4-63724365aeb1\") " pod="openshift-dns/node-resolver-nrxwb" Nov 02 22:52:32.404198 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.404145 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/60832943-bf1f-4685-be00-08152c423b08-serving-cert\") pod \"service-ca-operator-544d749c8c-p66wt\" (UID: \"60832943-bf1f-4685-be00-08152c423b08\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p66wt" Nov 02 22:52:32.404198 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.404163 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0237eeb1-8c85-475e-8580-6dec324de705-image-registry-private-configuration\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:32.404198 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.404173 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/46eae5da-cf0b-4663-bdeb-f8220fa08c05-image-registry-private-configuration\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:32.404198 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.404162 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/15789a0d-e030-45e0-a129-41de738ef767-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mbmf9\" (UID: \"15789a0d-e030-45e0-a129-41de738ef767\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbmf9" Nov 02 22:52:32.404371 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.404213 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cml47\" (UniqueName: \"kubernetes.io/projected/3a464bf3-77fa-4fcd-8593-327fa1f4a19a-kube-api-access-cml47\") pod \"node-ca-4mhgx\" (UID: \"3a464bf3-77fa-4fcd-8593-327fa1f4a19a\") " pod="openshift-image-registry/node-ca-4mhgx" Nov 02 22:52:32.404371 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.404243 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/668094b0-0e10-47aa-ba49-6944f7083bc6-cnibin\") pod \"multus-additional-cni-plugins-cs2mc\" (UID: \"668094b0-0e10-47aa-ba49-6944f7083bc6\") " pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:52:32.404371 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.404360 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-etc-kubernetes\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:52:32.404502 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.404391 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-sys\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:52:32.404502 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.404410 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0237eeb1-8c85-475e-8580-6dec324de705-ca-trust-extracted\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:32.404502 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.404452 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/00b95c29-7df8-4602-bd0b-06b6281d5dc1-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-cgpkk\" (UID: \"00b95c29-7df8-4602-bd0b-06b6281d5dc1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cgpkk" Nov 02 22:52:32.404502 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.404474 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f98166f-8051-4dbd-9856-db692932050b-data-volume\") pod \"insights-runtime-extractor-sv7kh\" (UID: \"9f98166f-8051-4dbd-9856-db692932050b\") " pod="openshift-insights/insights-runtime-extractor-sv7kh" Nov 02 22:52:32.404728 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.404521 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0237eeb1-8c85-475e-8580-6dec324de705-installation-pull-secrets\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:32.404728 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.404532 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0237eeb1-8c85-475e-8580-6dec324de705-ca-trust-extracted\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:32.404728 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.404548 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fd96552a-1a78-4fac-8faf-71d36f010905-dbus\") pod \"global-pull-secret-syncer-k4gfs\" (UID: \"fd96552a-1a78-4fac-8faf-71d36f010905\") " pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:52:32.404728 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.404560 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f98166f-8051-4dbd-9856-db692932050b-data-volume\") pod \"insights-runtime-extractor-sv7kh\" (UID: \"9f98166f-8051-4dbd-9856-db692932050b\") " pod="openshift-insights/insights-runtime-extractor-sv7kh" Nov 02 22:52:32.404728 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.404579 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-host-var-lib-cni-multus\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.404728 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.404615 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/79e2a74e-667b-4851-9e8a-759cb2a91f98-signing-key\") pod \"service-ca-5fffc97bdf-hrdqf\" (UID: \"79e2a74e-667b-4851-9e8a-759cb2a91f98\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hrdqf" Nov 02 22:52:32.404728 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.404652 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/46eae5da-cf0b-4663-bdeb-f8220fa08c05-installation-pull-secrets\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:32.404728 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.404703 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/668094b0-0e10-47aa-ba49-6944f7083bc6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cs2mc\" (UID: \"668094b0-0e10-47aa-ba49-6944f7083bc6\") " pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:52:32.404728 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.404693 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/00b95c29-7df8-4602-bd0b-06b6281d5dc1-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-cgpkk\" (UID: \"00b95c29-7df8-4602-bd0b-06b6281d5dc1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cgpkk" Nov 02 22:52:32.405144 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.404748 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0237eeb1-8c85-475e-8580-6dec324de705-installation-pull-secrets\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:32.405144 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.404761 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f7d1d7-88e4-41e0-bdd3-a43df363b408-service-ca-bundle\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:32.405144 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.404798 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-etc-sysconfig\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:52:32.405144 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.404820 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0237eeb1-8c85-475e-8580-6dec324de705-registry-certificates\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:32.405144 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.404851 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-run-netns\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.405144 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.404874 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/46eae5da-cf0b-4663-bdeb-f8220fa08c05-installation-pull-secrets\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:32.405144 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.404878 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-registry-tls\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:32.405144 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.404921 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f98166f-8051-4dbd-9856-db692932050b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sv7kh\" (UID: \"9f98166f-8051-4dbd-9856-db692932050b\") " pod="openshift-insights/insights-runtime-extractor-sv7kh" Nov 02 22:52:32.405144 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.404956 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cb1743a1-6e7b-4f99-80e6-197a6916580d-etc-selinux\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:52:32.405144 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.404999 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0237eeb1-8c85-475e-8580-6dec324de705-registry-certificates\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:32.405144 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.405008 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/79e2a74e-667b-4851-9e8a-759cb2a91f98-signing-key\") pod \"service-ca-5fffc97bdf-hrdqf\" (UID: \"79e2a74e-667b-4851-9e8a-759cb2a91f98\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hrdqf" Nov 02 22:52:32.405144 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.405003 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmk5v\" (UniqueName: \"kubernetes.io/projected/60832943-bf1f-4685-be00-08152c423b08-kube-api-access-pmk5v\") pod \"service-ca-operator-544d749c8c-p66wt\" (UID: \"60832943-bf1f-4685-be00-08152c423b08\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p66wt" Nov 02 22:52:32.405144 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.405084 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/668094b0-0e10-47aa-ba49-6944f7083bc6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cs2mc\" (UID: \"668094b0-0e10-47aa-ba49-6944f7083bc6\") " pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:52:32.405709 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.405165 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-multus-cni-dir\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.405709 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.405212 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d12c211b-f607-43b9-b621-12f061f109d9-trusted-ca-bundle\") pod \"periodic-gathering-lx8nq-mmb4p\" (UID: \"d12c211b-f607-43b9-b621-12f061f109d9\") " pod="openshift-insights/periodic-gathering-lx8nq-mmb4p" Nov 02 22:52:32.405709 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.405245 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx9mw\" (UniqueName: \"kubernetes.io/projected/d12c211b-f607-43b9-b621-12f061f109d9-kube-api-access-bx9mw\") pod \"periodic-gathering-lx8nq-mmb4p\" (UID: \"d12c211b-f607-43b9-b621-12f061f109d9\") " pod="openshift-insights/periodic-gathering-lx8nq-mmb4p" Nov 02 22:52:32.405709 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.405273 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f8b41ab-cbec-4f72-a5d9-853953748c0b-iptables-alerter-script\") pod \"iptables-alerter-92hm7\" (UID: \"4f8b41ab-cbec-4f72-a5d9-853953748c0b\") " pod="openshift-network-operator/iptables-alerter-92hm7" Nov 02 22:52:32.405709 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.405304 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-run-openvswitch\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.405709 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.405328 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-cni-netd\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.405709 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.405376 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89zw7\" (UniqueName: \"kubernetes.io/projected/9c4b6fbd-74cf-41c6-bbae-92748ac25e1e-kube-api-access-89zw7\") pod \"console-operator-7cd56b8dbc-c2gxc\" (UID: \"9c4b6fbd-74cf-41c6-bbae-92748ac25e1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c2gxc" Nov 02 22:52:32.405709 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.405397 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6c7074b-4caa-43fb-9fca-14ebb5abf958-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kkzvx\" (UID: \"b6c7074b-4caa-43fb-9fca-14ebb5abf958\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkzvx" Nov 02 22:52:32.405709 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.405423 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8fbcc658-b377-4da4-bf1d-e6274dfd805b-agent-certs\") pod \"konnectivity-agent-zqssw\" (UID: \"8fbcc658-b377-4da4-bf1d-e6274dfd805b\") " pod="kube-system/konnectivity-agent-zqssw" Nov 02 22:52:32.405709 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.405492 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d12c211b-f607-43b9-b621-12f061f109d9-trusted-ca-bundle\") pod \"periodic-gathering-lx8nq-mmb4p\" (UID: \"d12c211b-f607-43b9-b621-12f061f109d9\") " pod="openshift-insights/periodic-gathering-lx8nq-mmb4p" Nov 02 22:52:32.405709 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.405541 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkqkc\" (UniqueName: \"kubernetes.io/projected/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-kube-api-access-kkqkc\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.405709 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.405565 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-host-run-multus-certs\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.405709 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.405583 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/668094b0-0e10-47aa-ba49-6944f7083bc6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cs2mc\" (UID: \"668094b0-0e10-47aa-ba49-6944f7083bc6\") " pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:52:32.405709 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.405610 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f8b41ab-cbec-4f72-a5d9-853953748c0b-iptables-alerter-script\") pod \"iptables-alerter-92hm7\" (UID: \"4f8b41ab-cbec-4f72-a5d9-853953748c0b\") " pod="openshift-network-operator/iptables-alerter-92hm7" Nov 02 22:52:32.405709 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.405685 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8fbcc658-b377-4da4-bf1d-e6274dfd805b-agent-certs\") pod \"konnectivity-agent-zqssw\" (UID: \"8fbcc658-b377-4da4-bf1d-e6274dfd805b\") " pod="kube-system/konnectivity-agent-zqssw" Nov 02 22:52:32.405709 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.405694 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cb1743a1-6e7b-4f99-80e6-197a6916580d-sys-fs\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:52:32.406467 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.405723 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fd96552a-1a78-4fac-8faf-71d36f010905-kubelet-config\") pod \"global-pull-secret-syncer-k4gfs\" (UID: \"fd96552a-1a78-4fac-8faf-71d36f010905\") " pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:52:32.406467 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.405754 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-etc-modprobe-d\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:52:32.406467 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.405772 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b3753c7-7634-4209-8760-7a52da534549-tmp\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:52:32.406467 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.405844 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/00b95c29-7df8-4602-bd0b-06b6281d5dc1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-cgpkk\" (UID: \"00b95c29-7df8-4602-bd0b-06b6281d5dc1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cgpkk" Nov 02 22:52:32.406467 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.405868 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-slash\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.406467 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.405878 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b3753c7-7634-4209-8760-7a52da534549-tmp\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:52:32.406467 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.405914 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws5j7\" (UniqueName: \"kubernetes.io/projected/b858466f-3dcd-419c-9495-4dbb89e8d51b-kube-api-access-ws5j7\") pod \"insights-operator-7c4986d449-xszmw\" (UID: \"b858466f-3dcd-419c-9495-4dbb89e8d51b\") " pod="openshift-insights/insights-operator-7c4986d449-xszmw" Nov 02 22:52:32.406467 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.406052 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-bound-sa-token\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:32.406467 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.406071 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-host-var-lib-kubelet\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.406467 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.406086 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9f98166f-8051-4dbd-9856-db692932050b-crio-socket\") pod \"insights-runtime-extractor-sv7kh\" (UID: \"9f98166f-8051-4dbd-9856-db692932050b\") " pod="openshift-insights/insights-runtime-extractor-sv7kh" Nov 02 22:52:32.406467 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.406104 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/df418d6f-b9d3-4019-967c-9511be123e1f-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ht25s\" (UID: \"df418d6f-b9d3-4019-967c-9511be123e1f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ht25s" Nov 02 22:52:32.406467 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.406204 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d12c211b-f607-43b9-b621-12f061f109d9-service-ca-bundle\") pod \"periodic-gathering-lx8nq-mmb4p\" (UID: \"d12c211b-f607-43b9-b621-12f061f109d9\") " pod="openshift-insights/periodic-gathering-lx8nq-mmb4p" Nov 02 22:52:32.406467 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.406249 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3575a78-aad1-41e0-bad4-63724365aeb1-tmp-dir\") pod \"node-resolver-nrxwb\" (UID: \"d3575a78-aad1-41e0-bad4-63724365aeb1\") " pod="openshift-dns/node-resolver-nrxwb" Nov 02 22:52:32.406467 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.406317 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/15789a0d-e030-45e0-a129-41de738ef767-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-mbmf9\" (UID: \"15789a0d-e030-45e0-a129-41de738ef767\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbmf9" Nov 02 22:52:32.406467 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.406346 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqdk5\" (UniqueName: \"kubernetes.io/projected/0d008a27-d5ba-49cb-b3fb-abdecd723abd-kube-api-access-fqdk5\") pod \"volume-data-source-validator-58c9f5f46d-867x9\" (UID: \"0d008a27-d5ba-49cb-b3fb-abdecd723abd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-867x9" Nov 02 22:52:32.406467 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.406323 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/df418d6f-b9d3-4019-967c-9511be123e1f-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ht25s\" (UID: \"df418d6f-b9d3-4019-967c-9511be123e1f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ht25s" Nov 02 22:52:32.406467 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.406363 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzxgm\" (UniqueName: \"kubernetes.io/projected/6b3753c7-7634-4209-8760-7a52da534549-kube-api-access-wzxgm\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:52:32.406467 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.406384 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-hostroot\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.406467 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.406323 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3575a78-aad1-41e0-bad4-63724365aeb1-tmp-dir\") pod \"node-resolver-nrxwb\" (UID: \"d3575a78-aad1-41e0-bad4-63724365aeb1\") " pod="openshift-dns/node-resolver-nrxwb" Nov 02 22:52:32.406467 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.406407 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d3575a78-aad1-41e0-bad4-63724365aeb1-hosts-file\") pod \"node-resolver-nrxwb\" (UID: \"d3575a78-aad1-41e0-bad4-63724365aeb1\") " pod="openshift-dns/node-resolver-nrxwb" Nov 02 22:52:32.406467 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.406406 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d12c211b-f607-43b9-b621-12f061f109d9-service-ca-bundle\") pod \"periodic-gathering-lx8nq-mmb4p\" (UID: \"d12c211b-f607-43b9-b621-12f061f109d9\") " pod="openshift-insights/periodic-gathering-lx8nq-mmb4p" Nov 02 22:52:32.406467 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.406441 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a464bf3-77fa-4fcd-8593-327fa1f4a19a-host\") pod \"node-ca-4mhgx\" (UID: \"3a464bf3-77fa-4fcd-8593-327fa1f4a19a\") " pod="openshift-image-registry/node-ca-4mhgx" Nov 02 22:52:32.406467 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.406463 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-stats-auth\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:32.406467 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.406478 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-systemd-units\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.407518 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.406507 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/15789a0d-e030-45e0-a129-41de738ef767-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-mbmf9\" (UID: \"15789a0d-e030-45e0-a129-41de738ef767\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbmf9" Nov 02 22:52:32.407518 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.406517 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-cni-bin\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.407518 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.406537 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-etc-sysctl-d\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:52:32.407518 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.406629 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/668094b0-0e10-47aa-ba49-6944f7083bc6-cni-binary-copy\") pod \"multus-additional-cni-plugins-cs2mc\" (UID: \"668094b0-0e10-47aa-ba49-6944f7083bc6\") " pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:52:32.407518 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.406645 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cb1743a1-6e7b-4f99-80e6-197a6916580d-registration-dir\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:52:32.407518 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.406655 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-stats-auth\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:32.407518 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.406659 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-log-socket\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.407518 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.406682 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-run\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:52:32.407518 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.406700 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwnxj\" (UniqueName: \"kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-kube-api-access-nwnxj\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:32.407518 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.406716 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f98166f-8051-4dbd-9856-db692932050b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sv7kh\" (UID: \"9f98166f-8051-4dbd-9856-db692932050b\") " pod="openshift-insights/insights-runtime-extractor-sv7kh" Nov 02 22:52:32.407518 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.406732 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24smc\" (UniqueName: \"kubernetes.io/projected/fbcab4d0-9dd6-4682-be01-c4f589697e95-kube-api-access-24smc\") pod \"kas-connection-checker-685895c7c9-pxpxp\" (UID: \"fbcab4d0-9dd6-4682-be01-c4f589697e95\") " pod="kube-system/kas-connection-checker-685895c7c9-pxpxp" Nov 02 22:52:32.407518 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.406748 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac4a7fda-863e-4fb1-a98c-9b325a8c0df7-cert\") pod \"ingress-canary-h9lck\" (UID: \"ac4a7fda-863e-4fb1-a98c-9b325a8c0df7\") " pod="openshift-ingress-canary/ingress-canary-h9lck" Nov 02 22:52:32.407518 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.406766 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/668094b0-0e10-47aa-ba49-6944f7083bc6-cni-binary-copy\") pod \"multus-additional-cni-plugins-cs2mc\" (UID: \"668094b0-0e10-47aa-ba49-6944f7083bc6\") " pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:52:32.407518 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.406794 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j4hf\" (UniqueName: \"kubernetes.io/projected/50d13ae8-4ab5-4ca7-a11a-ad632da19669-kube-api-access-5j4hf\") pod \"migrator-66df5d496b-f62lt\" (UID: \"50d13ae8-4ab5-4ca7-a11a-ad632da19669\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-f62lt" Nov 02 22:52:32.407518 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.406822 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3a464bf3-77fa-4fcd-8593-327fa1f4a19a-serviceca\") pod \"node-ca-4mhgx\" (UID: \"3a464bf3-77fa-4fcd-8593-327fa1f4a19a\") " pod="openshift-image-registry/node-ca-4mhgx" Nov 02 22:52:32.407518 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.406865 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-etc-openvswitch\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.407518 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.406915 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee52c167-6361-4adb-9899-93ca4c2128b8-cni-binary-copy\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.407518 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.406974 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6flqc\" (UniqueName: \"kubernetes.io/projected/0b6b45c3-5275-4877-a84b-15e602100c40-kube-api-access-6flqc\") pod \"migrator-66df5d496b-8mxfd\" (UID: \"0b6b45c3-5275-4877-a84b-15e602100c40\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8mxfd" Nov 02 22:52:32.407518 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.406981 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac4a7fda-863e-4fb1-a98c-9b325a8c0df7-cert\") pod \"ingress-canary-h9lck\" (UID: \"ac4a7fda-863e-4fb1-a98c-9b325a8c0df7\") " pod="openshift-ingress-canary/ingress-canary-h9lck" Nov 02 22:52:32.407518 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.407100 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-etc-systemd\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:52:32.407518 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.407129 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3a464bf3-77fa-4fcd-8593-327fa1f4a19a-serviceca\") pod \"node-ca-4mhgx\" (UID: \"3a464bf3-77fa-4fcd-8593-327fa1f4a19a\") " pod="openshift-image-registry/node-ca-4mhgx" Nov 02 22:52:32.407518 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.407149 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/60832943-bf1f-4685-be00-08152c423b08-config\") pod \"service-ca-operator-544d749c8c-p66wt\" (UID: \"60832943-bf1f-4685-be00-08152c423b08\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p66wt" Nov 02 22:52:32.407518 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.407172 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee52c167-6361-4adb-9899-93ca4c2128b8-cni-binary-copy\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.407518 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.407225 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9c4b6fbd-74cf-41c6-bbae-92748ac25e1e-serving-cert\") pod \"console-operator-7cd56b8dbc-c2gxc\" (UID: \"9c4b6fbd-74cf-41c6-bbae-92748ac25e1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c2gxc" Nov 02 22:52:32.407518 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.407259 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6jlv\" (UniqueName: \"kubernetes.io/projected/fc69f07f-4001-4951-aff8-d3d6671cc5d4-kube-api-access-n6jlv\") pod \"network-metrics-daemon-9zqrz\" (UID: \"fc69f07f-4001-4951-aff8-d3d6671cc5d4\") " pod="openshift-multus/network-metrics-daemon-9zqrz" Nov 02 22:52:32.407518 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.407278 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d12c211b-f607-43b9-b621-12f061f109d9-archives-path\") pod \"periodic-gathering-lx8nq-mmb4p\" (UID: \"d12c211b-f607-43b9-b621-12f061f109d9\") " pod="openshift-insights/periodic-gathering-lx8nq-mmb4p" Nov 02 22:52:32.407518 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.407295 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/60832943-bf1f-4685-be00-08152c423b08-config\") pod \"service-ca-operator-544d749c8c-p66wt\" (UID: \"60832943-bf1f-4685-be00-08152c423b08\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p66wt" Nov 02 22:52:32.407518 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.407336 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b858466f-3dcd-419c-9495-4dbb89e8d51b-service-ca-bundle\") pod \"insights-operator-7c4986d449-xszmw\" (UID: \"b858466f-3dcd-419c-9495-4dbb89e8d51b\") " pod="openshift-insights/insights-operator-7c4986d449-xszmw" Nov 02 22:52:32.407518 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.407352 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b858466f-3dcd-419c-9495-4dbb89e8d51b-serving-cert\") pod \"insights-operator-7c4986d449-xszmw\" (UID: \"b858466f-3dcd-419c-9495-4dbb89e8d51b\") " pod="openshift-insights/insights-operator-7c4986d449-xszmw" Nov 02 22:52:32.407518 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.407368 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-system-cni-dir\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.407518 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.407386 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/46eae5da-cf0b-4663-bdeb-f8220fa08c05-trusted-ca\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:32.407518 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.407385 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9c4b6fbd-74cf-41c6-bbae-92748ac25e1e-serving-cert\") pod \"console-operator-7cd56b8dbc-c2gxc\" (UID: \"9c4b6fbd-74cf-41c6-bbae-92748ac25e1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c2gxc" Nov 02 22:52:32.407518 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.407418 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f8b41ab-cbec-4f72-a5d9-853953748c0b-host-slash\") pod \"iptables-alerter-92hm7\" (UID: \"4f8b41ab-cbec-4f72-a5d9-853953748c0b\") " pod="openshift-network-operator/iptables-alerter-92hm7" Nov 02 22:52:32.407518 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.407448 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t8ms\" (UniqueName: \"kubernetes.io/projected/ac4a7fda-863e-4fb1-a98c-9b325a8c0df7-kube-api-access-9t8ms\") pod \"ingress-canary-h9lck\" (UID: \"ac4a7fda-863e-4fb1-a98c-9b325a8c0df7\") " pod="openshift-ingress-canary/ingress-canary-h9lck" Nov 02 22:52:32.407518 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.407462 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d12c211b-f607-43b9-b621-12f061f109d9-archives-path\") pod \"periodic-gathering-lx8nq-mmb4p\" (UID: \"d12c211b-f607-43b9-b621-12f061f109d9\") " pod="openshift-insights/periodic-gathering-lx8nq-mmb4p" Nov 02 22:52:32.407518 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.407481 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85k6p\" (UniqueName: \"kubernetes.io/projected/9f98166f-8051-4dbd-9856-db692932050b-kube-api-access-85k6p\") pod \"insights-runtime-extractor-sv7kh\" (UID: \"9f98166f-8051-4dbd-9856-db692932050b\") " pod="openshift-insights/insights-runtime-extractor-sv7kh" Nov 02 22:52:32.407518 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.407536 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4shq\" (UniqueName: \"kubernetes.io/projected/668094b0-0e10-47aa-ba49-6944f7083bc6-kube-api-access-k4shq\") pod \"multus-additional-cni-plugins-cs2mc\" (UID: \"668094b0-0e10-47aa-ba49-6944f7083bc6\") " pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:52:32.407518 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.407541 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b858466f-3dcd-419c-9495-4dbb89e8d51b-serving-cert\") pod \"insights-operator-7c4986d449-xszmw\" (UID: \"b858466f-3dcd-419c-9495-4dbb89e8d51b\") " pod="openshift-insights/insights-operator-7c4986d449-xszmw" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.407606 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b858466f-3dcd-419c-9495-4dbb89e8d51b-service-ca-bundle\") pod \"insights-operator-7c4986d449-xszmw\" (UID: \"b858466f-3dcd-419c-9495-4dbb89e8d51b\") " pod="openshift-insights/insights-operator-7c4986d449-xszmw" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.407630 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cb1743a1-6e7b-4f99-80e6-197a6916580d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.407715 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-ovnkube-script-lib\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.407733 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/df418d6f-b9d3-4019-967c-9511be123e1f-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ht25s\" (UID: \"df418d6f-b9d3-4019-967c-9511be123e1f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ht25s" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.407792 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/79e2a74e-667b-4851-9e8a-759cb2a91f98-signing-cabundle\") pod \"service-ca-5fffc97bdf-hrdqf\" (UID: \"79e2a74e-667b-4851-9e8a-759cb2a91f98\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hrdqf" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.407955 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-ovnkube-script-lib\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.407988 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stv25\" (UniqueName: \"kubernetes.io/projected/42d598be-a6f1-486e-9ac3-2f5ece575f89-kube-api-access-stv25\") pod \"kas-connection-checker-685895c7c9-fdj8t\" (UID: \"42d598be-a6f1-486e-9ac3-2f5ece575f89\") " pod="kube-system/kas-connection-checker-685895c7c9-fdj8t" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.408002 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/79e2a74e-667b-4851-9e8a-759cb2a91f98-signing-cabundle\") pod \"service-ca-5fffc97bdf-hrdqf\" (UID: \"79e2a74e-667b-4851-9e8a-759cb2a91f98\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hrdqf" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.408028 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-multus-socket-dir-parent\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.408100 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/df418d6f-b9d3-4019-967c-9511be123e1f-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ht25s\" (UID: \"df418d6f-b9d3-4019-967c-9511be123e1f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ht25s" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.408126 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx46q\" (UniqueName: \"kubernetes.io/projected/79e2a74e-667b-4851-9e8a-759cb2a91f98-kube-api-access-lx46q\") pod \"service-ca-5fffc97bdf-hrdqf\" (UID: \"79e2a74e-667b-4851-9e8a-759cb2a91f98\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hrdqf" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.408153 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncmnm\" (UniqueName: \"kubernetes.io/projected/4b9cd8c7-228b-451a-afe4-3af7a3bf0a42-kube-api-access-ncmnm\") pod \"kas-connection-checker-685895c7c9-mtrcx\" (UID: \"4b9cd8c7-228b-451a-afe4-3af7a3bf0a42\") " pod="kube-system/kas-connection-checker-685895c7c9-mtrcx" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.408169 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-run-ovn-kubernetes\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.408186 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/46eae5da-cf0b-4663-bdeb-f8220fa08c05-ca-trust-extracted\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.408211 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cb1743a1-6e7b-4f99-80e6-197a6916580d-socket-dir\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.408237 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.408272 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/46eae5da-cf0b-4663-bdeb-f8220fa08c05-ca-trust-extracted\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.408269 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd96552a-1a78-4fac-8faf-71d36f010905-original-pull-secret\") pod \"global-pull-secret-syncer-k4gfs\" (UID: \"fd96552a-1a78-4fac-8faf-71d36f010905\") " pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.408288 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/46eae5da-cf0b-4663-bdeb-f8220fa08c05-trusted-ca\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.408310 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxxqc\" (UniqueName: \"kubernetes.io/projected/236450f1-d9fa-488b-9022-ab4f5517549f-kube-api-access-vxxqc\") pod \"network-check-source-6f78d59f99-6zkx2\" (UID: \"236450f1-d9fa-488b-9022-ab4f5517549f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6zkx2" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.408334 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rt8s\" (UniqueName: \"kubernetes.io/projected/cb1743a1-6e7b-4f99-80e6-197a6916580d-kube-api-access-6rt8s\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.408350 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-default-certificate\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.408376 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-lib-modules\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.408404 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-host-var-lib-cni-bin\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.408431 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkzvb\" (UniqueName: \"kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-kube-api-access-nkzvb\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.408457 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md7ml\" (UniqueName: \"kubernetes.io/projected/75f7d1d7-88e4-41e0-bdd3-a43df363b408-kube-api-access-md7ml\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.408496 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctsxp\" (UniqueName: \"kubernetes.io/projected/4f8b41ab-cbec-4f72-a5d9-853953748c0b-kube-api-access-ctsxp\") pod \"iptables-alerter-92hm7\" (UID: \"4f8b41ab-cbec-4f72-a5d9-853953748c0b\") " pod="openshift-network-operator/iptables-alerter-92hm7" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.408518 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-kubelet\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.408539 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd96552a-1a78-4fac-8faf-71d36f010905-original-pull-secret\") pod \"global-pull-secret-syncer-k4gfs\" (UID: \"fd96552a-1a78-4fac-8faf-71d36f010905\") " pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.408553 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-default-certificate\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.408564 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-host\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.408604 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9c4b6fbd-74cf-41c6-bbae-92748ac25e1e-trusted-ca\") pod \"console-operator-7cd56b8dbc-c2gxc\" (UID: \"9c4b6fbd-74cf-41c6-bbae-92748ac25e1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c2gxc" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.408646 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/105406c8-6800-4562-9651-962bee9f8d73-metrics-tls\") pod \"dns-default-pvc5f\" (UID: \"105406c8-6800-4562-9651-962bee9f8d73\") " pod="openshift-dns/dns-default-pvc5f" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.408767 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/46eae5da-cf0b-4663-bdeb-f8220fa08c05-registry-certificates\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.408796 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-metrics-certs\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.408880 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9c4b6fbd-74cf-41c6-bbae-92748ac25e1e-trusted-ca\") pod \"console-operator-7cd56b8dbc-c2gxc\" (UID: \"9c4b6fbd-74cf-41c6-bbae-92748ac25e1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c2gxc" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.408918 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-host-run-k8s-cni-cncf-io\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.408938 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/46eae5da-cf0b-4663-bdeb-f8220fa08c05-registry-certificates\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.408949 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-host-run-netns\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.408972 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ee52c167-6361-4adb-9899-93ca4c2128b8-multus-daemon-config\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.409001 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7qr5\" (UniqueName: \"kubernetes.io/projected/ee52c167-6361-4adb-9899-93ca4c2128b8-kube-api-access-q7qr5\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.409059 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnt9q\" (UniqueName: \"kubernetes.io/projected/15789a0d-e030-45e0-a129-41de738ef767-kube-api-access-jnt9q\") pod \"cluster-monitoring-operator-884576c57-mbmf9\" (UID: \"15789a0d-e030-45e0-a129-41de738ef767\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbmf9" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.409170 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ee52c167-6361-4adb-9899-93ca4c2128b8-multus-daemon-config\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.409174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.409198 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b858466f-3dcd-419c-9495-4dbb89e8d51b-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xszmw\" (UID: \"b858466f-3dcd-419c-9495-4dbb89e8d51b\") " pod="openshift-insights/insights-operator-7c4986d449-xszmw" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.409219 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-etc-sysctl-conf\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.409238 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-var-lib-kubelet\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.409281 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b6c7074b-4caa-43fb-9fca-14ebb5abf958-nginx-conf\") pod \"networking-console-plugin-86c478bb88-kkzvx\" (UID: \"b6c7074b-4caa-43fb-9fca-14ebb5abf958\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkzvx" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.409303 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-env-overrides\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.409329 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-ovn-node-metrics-cert\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.409345 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b858466f-3dcd-419c-9495-4dbb89e8d51b-snapshots\") pod \"insights-operator-7c4986d449-xszmw\" (UID: \"b858466f-3dcd-419c-9495-4dbb89e8d51b\") " pod="openshift-insights/insights-operator-7c4986d449-xszmw" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.409395 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b858466f-3dcd-419c-9495-4dbb89e8d51b-snapshots\") pod \"insights-operator-7c4986d449-xszmw\" (UID: \"b858466f-3dcd-419c-9495-4dbb89e8d51b\") " pod="openshift-insights/insights-operator-7c4986d449-xszmw" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.409430 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6b3753c7-7634-4209-8760-7a52da534549-etc-tuned\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.409427 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b858466f-3dcd-419c-9495-4dbb89e8d51b-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xszmw\" (UID: \"b858466f-3dcd-419c-9495-4dbb89e8d51b\") " pod="openshift-insights/insights-operator-7c4986d449-xszmw" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.409448 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b858466f-3dcd-419c-9495-4dbb89e8d51b-tmp\") pod \"insights-operator-7c4986d449-xszmw\" (UID: \"b858466f-3dcd-419c-9495-4dbb89e8d51b\") " pod="openshift-insights/insights-operator-7c4986d449-xszmw" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.409461 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-env-overrides\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.409488 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-multus-conf-dir\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.409501 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b6c7074b-4caa-43fb-9fca-14ebb5abf958-nginx-conf\") pod \"networking-console-plugin-86c478bb88-kkzvx\" (UID: \"b6c7074b-4caa-43fb-9fca-14ebb5abf958\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkzvx" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.409512 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6b3753c7-7634-4209-8760-7a52da534549-etc-tuned\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.409541 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/105406c8-6800-4562-9651-962bee9f8d73-tmp-dir\") pod \"dns-default-pvc5f\" (UID: \"105406c8-6800-4562-9651-962bee9f8d73\") " pod="openshift-dns/dns-default-pvc5f" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.409562 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b858466f-3dcd-419c-9495-4dbb89e8d51b-tmp\") pod \"insights-operator-7c4986d449-xszmw\" (UID: \"b858466f-3dcd-419c-9495-4dbb89e8d51b\") " pod="openshift-insights/insights-operator-7c4986d449-xszmw" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.409569 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f4r4\" (UniqueName: \"kubernetes.io/projected/105406c8-6800-4562-9651-962bee9f8d73-kube-api-access-9f4r4\") pod \"dns-default-pvc5f\" (UID: \"105406c8-6800-4562-9651-962bee9f8d73\") " pod="openshift-dns/dns-default-pvc5f" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.409602 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-bound-sa-token\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.409642 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/668094b0-0e10-47aa-ba49-6944f7083bc6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cs2mc\" (UID: \"668094b0-0e10-47aa-ba49-6944f7083bc6\") " pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.409663 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8fbcc658-b377-4da4-bf1d-e6274dfd805b-konnectivity-ca\") pod \"konnectivity-agent-zqssw\" (UID: \"8fbcc658-b377-4da4-bf1d-e6274dfd805b\") " pod="kube-system/konnectivity-agent-zqssw" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.409674 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/105406c8-6800-4562-9651-962bee9f8d73-tmp-dir\") pod \"dns-default-pvc5f\" (UID: \"105406c8-6800-4562-9651-962bee9f8d73\") " pod="openshift-dns/dns-default-pvc5f" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.409682 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-var-lib-openvswitch\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.409730 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0237eeb1-8c85-475e-8580-6dec324de705-trusted-ca\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.409759 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt847\" (UniqueName: \"kubernetes.io/projected/00b95c29-7df8-4602-bd0b-06b6281d5dc1-kube-api-access-zt847\") pod \"cluster-samples-operator-6ddb6c8b94-cgpkk\" (UID: \"00b95c29-7df8-4602-bd0b-06b6281d5dc1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cgpkk" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.409763 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-ovn-node-metrics-cert\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.409867 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8fbcc658-b377-4da4-bf1d-e6274dfd805b-konnectivity-ca\") pod \"konnectivity-agent-zqssw\" (UID: \"8fbcc658-b377-4da4-bf1d-e6274dfd805b\") " pod="kube-system/konnectivity-agent-zqssw" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.409874 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-cnibin\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.409934 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/668094b0-0e10-47aa-ba49-6944f7083bc6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cs2mc\" (UID: \"668094b0-0e10-47aa-ba49-6944f7083bc6\") " pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.409950 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/105406c8-6800-4562-9651-962bee9f8d73-config-volume\") pod \"dns-default-pvc5f\" (UID: \"105406c8-6800-4562-9651-962bee9f8d73\") " pod="openshift-dns/dns-default-pvc5f" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.409976 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/668094b0-0e10-47aa-ba49-6944f7083bc6-system-cni-dir\") pod \"multus-additional-cni-plugins-cs2mc\" (UID: \"668094b0-0e10-47aa-ba49-6944f7083bc6\") " pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.409997 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-run-systemd\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.410020 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-node-log\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.410035 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0237eeb1-8c85-475e-8580-6dec324de705-trusted-ca\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.410055 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7nrp\" (UniqueName: \"kubernetes.io/projected/e3a3bc57-6813-441c-bdc0-d39eb8214c61-kube-api-access-q7nrp\") pod \"network-check-target-rmgpt\" (UID: \"e3a3bc57-6813-441c-bdc0-d39eb8214c61\") " pod="openshift-network-diagnostics/network-check-target-rmgpt" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.410074 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9c4b6fbd-74cf-41c6-bbae-92748ac25e1e-config\") pod \"console-operator-7cd56b8dbc-c2gxc\" (UID: \"9c4b6fbd-74cf-41c6-bbae-92748ac25e1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c2gxc" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.410109 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs\") pod \"network-metrics-daemon-9zqrz\" (UID: \"fc69f07f-4001-4951-aff8-d3d6671cc5d4\") " pod="openshift-multus/network-metrics-daemon-9zqrz" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.410152 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-etc-kubernetes\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.410179 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-registry-tls\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.410190 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/105406c8-6800-4562-9651-962bee9f8d73-config-volume\") pod \"dns-default-pvc5f\" (UID: \"105406c8-6800-4562-9651-962bee9f8d73\") " pod="openshift-dns/dns-default-pvc5f" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.410193 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9c4b6fbd-74cf-41c6-bbae-92748ac25e1e-config\") pod \"console-operator-7cd56b8dbc-c2gxc\" (UID: \"9c4b6fbd-74cf-41c6-bbae-92748ac25e1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c2gxc" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.410217 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-run-ovn\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.410249 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-ovnkube-config\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.410270 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjljw\" (UniqueName: \"kubernetes.io/projected/df418d6f-b9d3-4019-967c-9511be123e1f-kube-api-access-jjljw\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ht25s\" (UID: \"df418d6f-b9d3-4019-967c-9511be123e1f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ht25s" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.410303 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cb1743a1-6e7b-4f99-80e6-197a6916580d-device-dir\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:52:32.411081 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.410424 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-ovnkube-config\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.459725 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.459691 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cml47\" (UniqueName: \"kubernetes.io/projected/3a464bf3-77fa-4fcd-8593-327fa1f4a19a-kube-api-access-cml47\") pod \"node-ca-4mhgx\" (UID: \"3a464bf3-77fa-4fcd-8593-327fa1f4a19a\") " pod="openshift-image-registry/node-ca-4mhgx" Nov 02 22:52:32.460981 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.460493 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89zw7\" (UniqueName: \"kubernetes.io/projected/9c4b6fbd-74cf-41c6-bbae-92748ac25e1e-kube-api-access-89zw7\") pod \"console-operator-7cd56b8dbc-c2gxc\" (UID: \"9c4b6fbd-74cf-41c6-bbae-92748ac25e1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c2gxc" Nov 02 22:52:32.460981 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.460541 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvg2k\" (UniqueName: \"kubernetes.io/projected/d3575a78-aad1-41e0-bad4-63724365aeb1-kube-api-access-vvg2k\") pod \"node-resolver-nrxwb\" (UID: \"d3575a78-aad1-41e0-bad4-63724365aeb1\") " pod="openshift-dns/node-resolver-nrxwb" Nov 02 22:52:32.460981 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.460679 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmk5v\" (UniqueName: \"kubernetes.io/projected/60832943-bf1f-4685-be00-08152c423b08-kube-api-access-pmk5v\") pod \"service-ca-operator-544d749c8c-p66wt\" (UID: \"60832943-bf1f-4685-be00-08152c423b08\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p66wt" Nov 02 22:52:32.460981 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.460807 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzxgm\" (UniqueName: \"kubernetes.io/projected/6b3753c7-7634-4209-8760-7a52da534549-kube-api-access-wzxgm\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:52:32.460981 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.460825 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws5j7\" (UniqueName: \"kubernetes.io/projected/b858466f-3dcd-419c-9495-4dbb89e8d51b-kube-api-access-ws5j7\") pod \"insights-operator-7c4986d449-xszmw\" (UID: \"b858466f-3dcd-419c-9495-4dbb89e8d51b\") " pod="openshift-insights/insights-operator-7c4986d449-xszmw" Nov 02 22:52:32.460981 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.460912 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkqkc\" (UniqueName: \"kubernetes.io/projected/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-kube-api-access-kkqkc\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.460981 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.460937 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqdk5\" (UniqueName: \"kubernetes.io/projected/0d008a27-d5ba-49cb-b3fb-abdecd723abd-kube-api-access-fqdk5\") pod \"volume-data-source-validator-58c9f5f46d-867x9\" (UID: \"0d008a27-d5ba-49cb-b3fb-abdecd723abd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-867x9" Nov 02 22:52:32.460981 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.460978 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx9mw\" (UniqueName: \"kubernetes.io/projected/d12c211b-f607-43b9-b621-12f061f109d9-kube-api-access-bx9mw\") pod \"periodic-gathering-lx8nq-mmb4p\" (UID: \"d12c211b-f607-43b9-b621-12f061f109d9\") " pod="openshift-insights/periodic-gathering-lx8nq-mmb4p" Nov 02 22:52:32.461430 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.461031 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-bound-sa-token\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:32.461430 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.461130 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6flqc\" (UniqueName: \"kubernetes.io/projected/0b6b45c3-5275-4877-a84b-15e602100c40-kube-api-access-6flqc\") pod \"migrator-66df5d496b-8mxfd\" (UID: \"0b6b45c3-5275-4877-a84b-15e602100c40\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8mxfd" Nov 02 22:52:32.461430 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.461156 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j4hf\" (UniqueName: \"kubernetes.io/projected/50d13ae8-4ab5-4ca7-a11a-ad632da19669-kube-api-access-5j4hf\") pod \"migrator-66df5d496b-f62lt\" (UID: \"50d13ae8-4ab5-4ca7-a11a-ad632da19669\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-f62lt" Nov 02 22:52:32.461430 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.461342 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24smc\" (UniqueName: \"kubernetes.io/projected/fbcab4d0-9dd6-4682-be01-c4f589697e95-kube-api-access-24smc\") pod \"kas-connection-checker-685895c7c9-pxpxp\" (UID: \"fbcab4d0-9dd6-4682-be01-c4f589697e95\") " pod="kube-system/kas-connection-checker-685895c7c9-pxpxp" Nov 02 22:52:32.462873 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.462855 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4shq\" (UniqueName: \"kubernetes.io/projected/668094b0-0e10-47aa-ba49-6944f7083bc6-kube-api-access-k4shq\") pod \"multus-additional-cni-plugins-cs2mc\" (UID: \"668094b0-0e10-47aa-ba49-6944f7083bc6\") " pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:52:32.463440 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.463421 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6jlv\" (UniqueName: \"kubernetes.io/projected/fc69f07f-4001-4951-aff8-d3d6671cc5d4-kube-api-access-n6jlv\") pod \"network-metrics-daemon-9zqrz\" (UID: \"fc69f07f-4001-4951-aff8-d3d6671cc5d4\") " pod="openshift-multus/network-metrics-daemon-9zqrz" Nov 02 22:52:32.463730 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.463711 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwnxj\" (UniqueName: \"kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-kube-api-access-nwnxj\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:32.467209 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.467188 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stv25\" (UniqueName: \"kubernetes.io/projected/42d598be-a6f1-486e-9ac3-2f5ece575f89-kube-api-access-stv25\") pod \"kas-connection-checker-685895c7c9-fdj8t\" (UID: \"42d598be-a6f1-486e-9ac3-2f5ece575f89\") " pod="kube-system/kas-connection-checker-685895c7c9-fdj8t" Nov 02 22:52:32.467370 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.467352 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkzvb\" (UniqueName: \"kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-kube-api-access-nkzvb\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:32.472117 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.472101 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rt8s\" (UniqueName: \"kubernetes.io/projected/cb1743a1-6e7b-4f99-80e6-197a6916580d-kube-api-access-6rt8s\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:52:32.474175 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.474155 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85k6p\" (UniqueName: \"kubernetes.io/projected/9f98166f-8051-4dbd-9856-db692932050b-kube-api-access-85k6p\") pod \"insights-runtime-extractor-sv7kh\" (UID: \"9f98166f-8051-4dbd-9856-db692932050b\") " pod="openshift-insights/insights-runtime-extractor-sv7kh" Nov 02 22:52:32.474634 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.474617 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md7ml\" (UniqueName: \"kubernetes.io/projected/75f7d1d7-88e4-41e0-bdd3-a43df363b408-kube-api-access-md7ml\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:32.474738 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.474720 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjljw\" (UniqueName: \"kubernetes.io/projected/df418d6f-b9d3-4019-967c-9511be123e1f-kube-api-access-jjljw\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ht25s\" (UID: \"df418d6f-b9d3-4019-967c-9511be123e1f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ht25s" Nov 02 22:52:32.476105 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.476081 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctsxp\" (UniqueName: \"kubernetes.io/projected/4f8b41ab-cbec-4f72-a5d9-853953748c0b-kube-api-access-ctsxp\") pod \"iptables-alerter-92hm7\" (UID: \"4f8b41ab-cbec-4f72-a5d9-853953748c0b\") " pod="openshift-network-operator/iptables-alerter-92hm7" Nov 02 22:52:32.476105 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.476095 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7qr5\" (UniqueName: \"kubernetes.io/projected/ee52c167-6361-4adb-9899-93ca4c2128b8-kube-api-access-q7qr5\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.476341 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.476325 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-bound-sa-token\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:32.476416 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.476403 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t8ms\" (UniqueName: \"kubernetes.io/projected/ac4a7fda-863e-4fb1-a98c-9b325a8c0df7-kube-api-access-9t8ms\") pod \"ingress-canary-h9lck\" (UID: \"ac4a7fda-863e-4fb1-a98c-9b325a8c0df7\") " pod="openshift-ingress-canary/ingress-canary-h9lck" Nov 02 22:52:32.476481 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.476459 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx46q\" (UniqueName: \"kubernetes.io/projected/79e2a74e-667b-4851-9e8a-759cb2a91f98-kube-api-access-lx46q\") pod \"service-ca-5fffc97bdf-hrdqf\" (UID: \"79e2a74e-667b-4851-9e8a-759cb2a91f98\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hrdqf" Nov 02 22:52:32.476562 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.476546 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt847\" (UniqueName: \"kubernetes.io/projected/00b95c29-7df8-4602-bd0b-06b6281d5dc1-kube-api-access-zt847\") pod \"cluster-samples-operator-6ddb6c8b94-cgpkk\" (UID: \"00b95c29-7df8-4602-bd0b-06b6281d5dc1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cgpkk" Nov 02 22:52:32.476956 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.476941 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f4r4\" (UniqueName: \"kubernetes.io/projected/105406c8-6800-4562-9651-962bee9f8d73-kube-api-access-9f4r4\") pod \"dns-default-pvc5f\" (UID: \"105406c8-6800-4562-9651-962bee9f8d73\") " pod="openshift-dns/dns-default-pvc5f" Nov 02 22:52:32.477158 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.477142 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7nrp\" (UniqueName: \"kubernetes.io/projected/e3a3bc57-6813-441c-bdc0-d39eb8214c61-kube-api-access-q7nrp\") pod \"network-check-target-rmgpt\" (UID: \"e3a3bc57-6813-441c-bdc0-d39eb8214c61\") " pod="openshift-network-diagnostics/network-check-target-rmgpt" Nov 02 22:52:32.477943 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.477921 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncmnm\" (UniqueName: \"kubernetes.io/projected/4b9cd8c7-228b-451a-afe4-3af7a3bf0a42-kube-api-access-ncmnm\") pod \"kas-connection-checker-685895c7c9-mtrcx\" (UID: \"4b9cd8c7-228b-451a-afe4-3af7a3bf0a42\") " pod="kube-system/kas-connection-checker-685895c7c9-mtrcx" Nov 02 22:52:32.478025 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.477931 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnt9q\" (UniqueName: \"kubernetes.io/projected/15789a0d-e030-45e0-a129-41de738ef767-kube-api-access-jnt9q\") pod \"cluster-monitoring-operator-884576c57-mbmf9\" (UID: \"15789a0d-e030-45e0-a129-41de738ef767\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbmf9" Nov 02 22:52:32.478025 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.477976 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxxqc\" (UniqueName: \"kubernetes.io/projected/236450f1-d9fa-488b-9022-ab4f5517549f-kube-api-access-vxxqc\") pod \"network-check-source-6f78d59f99-6zkx2\" (UID: \"236450f1-d9fa-488b-9022-ab4f5517549f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6zkx2" Nov 02 22:52:32.511556 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.511521 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-etc-systemd\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:52:32.511556 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.511556 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-system-cni-dir\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.511796 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.511573 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f8b41ab-cbec-4f72-a5d9-853953748c0b-host-slash\") pod \"iptables-alerter-92hm7\" (UID: \"4f8b41ab-cbec-4f72-a5d9-853953748c0b\") " pod="openshift-network-operator/iptables-alerter-92hm7" Nov 02 22:52:32.511796 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.511588 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cb1743a1-6e7b-4f99-80e6-197a6916580d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:52:32.511796 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.511639 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-multus-socket-dir-parent\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.511796 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.511655 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-run-ovn-kubernetes\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.511796 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.511657 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-etc-systemd\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:52:32.511796 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.511660 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-system-cni-dir\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.511796 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.511663 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f8b41ab-cbec-4f72-a5d9-853953748c0b-host-slash\") pod \"iptables-alerter-92hm7\" (UID: \"4f8b41ab-cbec-4f72-a5d9-853953748c0b\") " pod="openshift-network-operator/iptables-alerter-92hm7" Nov 02 22:52:32.511796 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.511672 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cb1743a1-6e7b-4f99-80e6-197a6916580d-socket-dir\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:52:32.511796 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.511717 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-multus-socket-dir-parent\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.511796 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.511736 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.511796 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.511740 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cb1743a1-6e7b-4f99-80e6-197a6916580d-socket-dir\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:52:32.511796 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.511699 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cb1743a1-6e7b-4f99-80e6-197a6916580d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:52:32.511796 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.511700 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-run-ovn-kubernetes\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.511796 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.511758 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-lib-modules\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:52:32.511796 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.511795 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.511818 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-host-var-lib-cni-bin\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.511843 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-kubelet\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.511863 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-host\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.511883 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/105406c8-6800-4562-9651-962bee9f8d73-metrics-tls\") pod \"dns-default-pvc5f\" (UID: \"105406c8-6800-4562-9651-962bee9f8d73\") " pod="openshift-dns/dns-default-pvc5f" Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.511922 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-host-var-lib-cni-bin\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.511929 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-metrics-certs\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.511934 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-kubelet\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.511935 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-lib-modules\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.511965 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-host-run-k8s-cni-cncf-io\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.511992 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-host-run-netns\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.511965 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-host\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:32.512008 5076 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512021 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-etc-sysctl-conf\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512042 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-host-run-netns\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512048 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-var-lib-kubelet\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512081 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-multus-conf-dir\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512046 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-host-run-k8s-cni-cncf-io\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:32.512011 5076 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512111 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-multus-conf-dir\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512097 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-var-lib-kubelet\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:32.512103 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/105406c8-6800-4562-9651-962bee9f8d73-metrics-tls podName:105406c8-6800-4562-9651-962bee9f8d73 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:33.012079948 +0000 UTC m=+1.980611733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/105406c8-6800-4562-9651-962bee9f8d73-metrics-tls") pod "dns-default-pvc5f" (UID: "105406c8-6800-4562-9651-962bee9f8d73") : secret "dns-default-metrics-tls" not found Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512141 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-etc-sysctl-conf\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:32.512188 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-metrics-certs podName:75f7d1d7-88e4-41e0-bdd3-a43df363b408 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:33.012174757 +0000 UTC m=+1.980706523 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-metrics-certs") pod "router-default-896ccfb6c-q4rg5" (UID: "75f7d1d7-88e4-41e0-bdd3-a43df363b408") : secret "router-metrics-certs-default" not found Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512202 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-var-lib-openvswitch\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512219 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-cnibin\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512236 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/668094b0-0e10-47aa-ba49-6944f7083bc6-system-cni-dir\") pod \"multus-additional-cni-plugins-cs2mc\" (UID: \"668094b0-0e10-47aa-ba49-6944f7083bc6\") " pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512250 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-run-systemd\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512239 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-var-lib-openvswitch\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512285 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/668094b0-0e10-47aa-ba49-6944f7083bc6-system-cni-dir\") pod \"multus-additional-cni-plugins-cs2mc\" (UID: \"668094b0-0e10-47aa-ba49-6944f7083bc6\") " pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512291 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-cnibin\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512309 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-node-log\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512314 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-run-systemd\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512328 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs\") pod \"network-metrics-daemon-9zqrz\" (UID: \"fc69f07f-4001-4951-aff8-d3d6671cc5d4\") " pod="openshift-multus/network-metrics-daemon-9zqrz" Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512344 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-etc-kubernetes\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512371 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-etc-kubernetes\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512373 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-node-log\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512388 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-registry-tls\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512409 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-run-ovn\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:32.512426 5076 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512426 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cb1743a1-6e7b-4f99-80e6-197a6916580d-device-dir\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512459 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cb1743a1-6e7b-4f99-80e6-197a6916580d-device-dir\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:52:32.512436 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:32.512467 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs podName:fc69f07f-4001-4951-aff8-d3d6671cc5d4 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:33.012453937 +0000 UTC m=+1.980985727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs") pod "network-metrics-daemon-9zqrz" (UID: "fc69f07f-4001-4951-aff8-d3d6671cc5d4") : secret "metrics-daemon-secret" not found Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512492 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-run-ovn\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512498 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/15789a0d-e030-45e0-a129-41de738ef767-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mbmf9\" (UID: \"15789a0d-e030-45e0-a129-41de738ef767\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbmf9" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512528 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/668094b0-0e10-47aa-ba49-6944f7083bc6-cnibin\") pod \"multus-additional-cni-plugins-cs2mc\" (UID: \"668094b0-0e10-47aa-ba49-6944f7083bc6\") " pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512550 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-etc-kubernetes\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512568 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-sys\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:32.512574 5076 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512591 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fd96552a-1a78-4fac-8faf-71d36f010905-dbus\") pod \"global-pull-secret-syncer-k4gfs\" (UID: \"fd96552a-1a78-4fac-8faf-71d36f010905\") " pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512615 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-host-var-lib-cni-multus\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:32.512624 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15789a0d-e030-45e0-a129-41de738ef767-cluster-monitoring-operator-tls podName:15789a0d-e030-45e0-a129-41de738ef767 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:33.012612227 +0000 UTC m=+1.981144004 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/15789a0d-e030-45e0-a129-41de738ef767-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-mbmf9" (UID: "15789a0d-e030-45e0-a129-41de738ef767") : secret "cluster-monitoring-operator-tls" not found Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512635 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-etc-kubernetes\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512647 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f7d1d7-88e4-41e0-bdd3-a43df363b408-service-ca-bundle\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512615 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/668094b0-0e10-47aa-ba49-6944f7083bc6-cnibin\") pod \"multus-additional-cni-plugins-cs2mc\" (UID: \"668094b0-0e10-47aa-ba49-6944f7083bc6\") " pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512655 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-host-var-lib-cni-multus\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512659 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-sys\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512664 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-etc-sysconfig\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512669 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fd96552a-1a78-4fac-8faf-71d36f010905-dbus\") pod \"global-pull-secret-syncer-k4gfs\" (UID: \"fd96552a-1a78-4fac-8faf-71d36f010905\") " pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512703 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-etc-sysconfig\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:32.512732 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75f7d1d7-88e4-41e0-bdd3-a43df363b408-service-ca-bundle podName:75f7d1d7-88e4-41e0-bdd3-a43df363b408 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:33.012718925 +0000 UTC m=+1.981250715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/75f7d1d7-88e4-41e0-bdd3-a43df363b408-service-ca-bundle") pod "router-default-896ccfb6c-q4rg5" (UID: "75f7d1d7-88e4-41e0-bdd3-a43df363b408") : configmap references non-existent config key: service-ca.crt Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512704 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-run-netns\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512745 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-run-netns\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512763 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-registry-tls\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512764 5076 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512781 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f98166f-8051-4dbd-9856-db692932050b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sv7kh\" (UID: \"9f98166f-8051-4dbd-9856-db692932050b\") " pod="openshift-insights/insights-runtime-extractor-sv7kh" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512802 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cb1743a1-6e7b-4f99-80e6-197a6916580d-etc-selinux\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512817 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-multus-cni-dir\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512834 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-run-openvswitch\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512855 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-cni-netd\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512883 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6c7074b-4caa-43fb-9fca-14ebb5abf958-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kkzvx\" (UID: \"b6c7074b-4caa-43fb-9fca-14ebb5abf958\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkzvx" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512915 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-multus-cni-dir\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512927 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cb1743a1-6e7b-4f99-80e6-197a6916580d-etc-selinux\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512929 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-host-run-multus-certs\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512962 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/668094b0-0e10-47aa-ba49-6944f7083bc6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cs2mc\" (UID: \"668094b0-0e10-47aa-ba49-6944f7083bc6\") " pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512971 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-host-run-multus-certs\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512986 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cb1743a1-6e7b-4f99-80e6-197a6916580d-sys-fs\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:32.512995 5076 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.513014 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fd96552a-1a78-4fac-8faf-71d36f010905-kubelet-config\") pod \"global-pull-secret-syncer-k4gfs\" (UID: \"fd96552a-1a78-4fac-8faf-71d36f010905\") " pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.513030 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cb1743a1-6e7b-4f99-80e6-197a6916580d-sys-fs\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.513037 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-cni-netd\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.512918 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-run-openvswitch\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.513042 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-etc-modprobe-d\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.513078 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/00b95c29-7df8-4602-bd0b-06b6281d5dc1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-cgpkk\" (UID: \"00b95c29-7df8-4602-bd0b-06b6281d5dc1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cgpkk" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:32.513090 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6c7074b-4caa-43fb-9fca-14ebb5abf958-networking-console-plugin-cert podName:b6c7074b-4caa-43fb-9fca-14ebb5abf958 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:33.013077247 +0000 UTC m=+1.981609022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b6c7074b-4caa-43fb-9fca-14ebb5abf958-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kkzvx" (UID: "b6c7074b-4caa-43fb-9fca-14ebb5abf958") : secret "networking-console-plugin-cert" not found Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.513126 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fd96552a-1a78-4fac-8faf-71d36f010905-kubelet-config\") pod \"global-pull-secret-syncer-k4gfs\" (UID: \"fd96552a-1a78-4fac-8faf-71d36f010905\") " pod="kube-system/global-pull-secret-syncer-k4gfs" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.513142 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-slash\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:32.513130 5076 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.513171 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-host-var-lib-kubelet\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.513191 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-slash\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.513195 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9f98166f-8051-4dbd-9856-db692932050b-crio-socket\") pod \"insights-runtime-extractor-sv7kh\" (UID: \"9f98166f-8051-4dbd-9856-db692932050b\") " pod="openshift-insights/insights-runtime-extractor-sv7kh" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.513164 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/668094b0-0e10-47aa-ba49-6944f7083bc6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cs2mc\" (UID: \"668094b0-0e10-47aa-ba49-6944f7083bc6\") " pod="openshift-multus/multus-additional-cni-plugins-cs2mc" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:32.513209 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00b95c29-7df8-4602-bd0b-06b6281d5dc1-samples-operator-tls podName:00b95c29-7df8-4602-bd0b-06b6281d5dc1 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:33.013195093 +0000 UTC m=+1.981726870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/00b95c29-7df8-4602-bd0b-06b6281d5dc1-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-cgpkk" (UID: "00b95c29-7df8-4602-bd0b-06b6281d5dc1") : secret "samples-operator-tls" not found Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.513222 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-etc-modprobe-d\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.513239 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9f98166f-8051-4dbd-9856-db692932050b-crio-socket\") pod \"insights-runtime-extractor-sv7kh\" (UID: \"9f98166f-8051-4dbd-9856-db692932050b\") " pod="openshift-insights/insights-runtime-extractor-sv7kh" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.513258 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-hostroot\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.513272 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-host-var-lib-kubelet\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.513282 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d3575a78-aad1-41e0-bad4-63724365aeb1-hosts-file\") pod \"node-resolver-nrxwb\" (UID: \"d3575a78-aad1-41e0-bad4-63724365aeb1\") " pod="openshift-dns/node-resolver-nrxwb" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.513327 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a464bf3-77fa-4fcd-8593-327fa1f4a19a-host\") pod \"node-ca-4mhgx\" (UID: \"3a464bf3-77fa-4fcd-8593-327fa1f4a19a\") " pod="openshift-image-registry/node-ca-4mhgx" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.513351 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-systemd-units\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.513355 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ee52c167-6361-4adb-9899-93ca4c2128b8-hostroot\") pod \"multus-njn94\" (UID: \"ee52c167-6361-4adb-9899-93ca4c2128b8\") " pod="openshift-multus/multus-njn94" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.513374 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-cni-bin\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.513395 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f98166f-8051-4dbd-9856-db692932050b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sv7kh\" (UID: \"9f98166f-8051-4dbd-9856-db692932050b\") " pod="openshift-insights/insights-runtime-extractor-sv7kh" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.513394 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-etc-sysctl-d\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.513430 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-systemd-units\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.513425 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a464bf3-77fa-4fcd-8593-327fa1f4a19a-host\") pod \"node-ca-4mhgx\" (UID: \"3a464bf3-77fa-4fcd-8593-327fa1f4a19a\") " pod="openshift-image-registry/node-ca-4mhgx" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.513399 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d3575a78-aad1-41e0-bad4-63724365aeb1-hosts-file\") pod \"node-resolver-nrxwb\" (UID: \"d3575a78-aad1-41e0-bad4-63724365aeb1\") " pod="openshift-dns/node-resolver-nrxwb" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.513459 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-host-cni-bin\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.513460 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cb1743a1-6e7b-4f99-80e6-197a6916580d-registration-dir\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.513502 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-etc-sysctl-d\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.513509 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cb1743a1-6e7b-4f99-80e6-197a6916580d-registration-dir\") pod \"aws-ebs-csi-driver-node-hqrsv\" (UID: \"cb1743a1-6e7b-4f99-80e6-197a6916580d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqrsv" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.513517 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-log-socket\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.513542 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-run\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.513548 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-log-socket\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.513586 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f98166f-8051-4dbd-9856-db692932050b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sv7kh\" (UID: \"9f98166f-8051-4dbd-9856-db692932050b\") " pod="openshift-insights/insights-runtime-extractor-sv7kh" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.513616 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-etc-openvswitch\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.513652 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6b3753c7-7634-4209-8760-7a52da534549-run\") pod \"tuned-zjn7n\" (UID: \"6b3753c7-7634-4209-8760-7a52da534549\") " pod="openshift-cluster-node-tuning-operator/tuned-zjn7n" Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:32.513738 5076 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:32.513781 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f98166f-8051-4dbd-9856-db692932050b-insights-runtime-extractor-tls podName:9f98166f-8051-4dbd-9856-db692932050b nodeName:}" failed. No retries permitted until 2026-11-02 22:52:33.013766276 +0000 UTC m=+1.982298051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f98166f-8051-4dbd-9856-db692932050b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sv7kh" (UID: "9f98166f-8051-4dbd-9856-db692932050b") : secret "insights-runtime-extractor-tls" not found Nov 02 22:52:32.514237 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.513739 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49-etc-openvswitch\") pod \"ovnkube-node-bp6bf\" (UID: \"324d3fbb-66cd-4e3f-8cc1-f65ce06c0d49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:32.516376 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.515123 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-registry-tls\") pod \"image-registry-7b5b8cd649-z4rbs\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:32.516376 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.515156 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-registry-tls\") pod \"image-registry-777f69544d-d75n6\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:32.551346 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.551279 5076 scope.go:122] "RemoveContainer" containerID="61bc91e4752e543b447d5adcc7109202c5002dfd3c1c92a91e17d4830f0014e5" Nov 02 22:52:32.551573 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.551553 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:32.554183 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.554165 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:32.768463 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:32.768426 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4mhgx_3a464bf3-77fa-4fcd-8593-327fa1f4a19a/node-ca/0.log" Nov 02 22:52:33.013922 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:33.013817 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-777f69544d-d75n6"] Nov 02 22:52:33.016737 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:33.016711 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b5b8cd649-z4rbs"] Nov 02 22:52:33.019006 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:33.018987 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6c7074b-4caa-43fb-9fca-14ebb5abf958-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kkzvx\" (UID: \"b6c7074b-4caa-43fb-9fca-14ebb5abf958\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkzvx" Nov 02 22:52:33.019104 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:33.019028 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/00b95c29-7df8-4602-bd0b-06b6281d5dc1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-cgpkk\" (UID: \"00b95c29-7df8-4602-bd0b-06b6281d5dc1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cgpkk" Nov 02 22:52:33.019104 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:33.019073 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f98166f-8051-4dbd-9856-db692932050b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sv7kh\" (UID: \"9f98166f-8051-4dbd-9856-db692932050b\") " pod="openshift-insights/insights-runtime-extractor-sv7kh" Nov 02 22:52:33.019227 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:33.019133 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/105406c8-6800-4562-9651-962bee9f8d73-metrics-tls\") pod \"dns-default-pvc5f\" (UID: \"105406c8-6800-4562-9651-962bee9f8d73\") " pod="openshift-dns/dns-default-pvc5f" Nov 02 22:52:33.019227 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:33.019164 5076 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 22:52:33.019227 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:33.019186 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-metrics-certs\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:33.019227 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:33.019168 5076 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 22:52:33.019227 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:33.019227 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00b95c29-7df8-4602-bd0b-06b6281d5dc1-samples-operator-tls podName:00b95c29-7df8-4602-bd0b-06b6281d5dc1 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:34.019214419 +0000 UTC m=+2.987746181 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/00b95c29-7df8-4602-bd0b-06b6281d5dc1-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-cgpkk" (UID: "00b95c29-7df8-4602-bd0b-06b6281d5dc1") : secret "samples-operator-tls" not found Nov 02 22:52:33.019430 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:33.019267 5076 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 22:52:33.019430 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:33.019273 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs\") pod \"network-metrics-daemon-9zqrz\" (UID: \"fc69f07f-4001-4951-aff8-d3d6671cc5d4\") " pod="openshift-multus/network-metrics-daemon-9zqrz" Nov 02 22:52:33.019430 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:33.019287 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6c7074b-4caa-43fb-9fca-14ebb5abf958-networking-console-plugin-cert podName:b6c7074b-4caa-43fb-9fca-14ebb5abf958 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:34.019270247 +0000 UTC m=+2.987802026 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b6c7074b-4caa-43fb-9fca-14ebb5abf958-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kkzvx" (UID: "b6c7074b-4caa-43fb-9fca-14ebb5abf958") : secret "networking-console-plugin-cert" not found Nov 02 22:52:33.019430 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:33.019286 5076 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 22:52:33.019430 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:33.019223 5076 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 22:52:33.019430 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:33.019312 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f98166f-8051-4dbd-9856-db692932050b-insights-runtime-extractor-tls podName:9f98166f-8051-4dbd-9856-db692932050b nodeName:}" failed. No retries permitted until 2026-11-02 22:52:34.019297998 +0000 UTC m=+2.987829784 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f98166f-8051-4dbd-9856-db692932050b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sv7kh" (UID: "9f98166f-8051-4dbd-9856-db692932050b") : secret "insights-runtime-extractor-tls" not found Nov 02 22:52:33.019430 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:33.019322 5076 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 22:52:33.019430 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:33.019334 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/105406c8-6800-4562-9651-962bee9f8d73-metrics-tls podName:105406c8-6800-4562-9651-962bee9f8d73 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:34.019322832 +0000 UTC m=+2.987854598 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/105406c8-6800-4562-9651-962bee9f8d73-metrics-tls") pod "dns-default-pvc5f" (UID: "105406c8-6800-4562-9651-962bee9f8d73") : secret "dns-default-metrics-tls" not found Nov 02 22:52:33.019430 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:33.019352 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-metrics-certs podName:75f7d1d7-88e4-41e0-bdd3-a43df363b408 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:34.019344693 +0000 UTC m=+2.987876454 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-metrics-certs") pod "router-default-896ccfb6c-q4rg5" (UID: "75f7d1d7-88e4-41e0-bdd3-a43df363b408") : secret "router-metrics-certs-default" not found Nov 02 22:52:33.019430 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:33.019366 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs podName:fc69f07f-4001-4951-aff8-d3d6671cc5d4 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:34.01936109 +0000 UTC m=+2.987892852 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs") pod "network-metrics-daemon-9zqrz" (UID: "fc69f07f-4001-4951-aff8-d3d6671cc5d4") : secret "metrics-daemon-secret" not found Nov 02 22:52:33.019430 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:33.019389 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/15789a0d-e030-45e0-a129-41de738ef767-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mbmf9\" (UID: \"15789a0d-e030-45e0-a129-41de738ef767\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbmf9" Nov 02 22:52:33.019430 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:33.019415 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f7d1d7-88e4-41e0-bdd3-a43df363b408-service-ca-bundle\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:33.019867 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:33.019498 5076 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 22:52:33.019867 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:33.019505 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75f7d1d7-88e4-41e0-bdd3-a43df363b408-service-ca-bundle podName:75f7d1d7-88e4-41e0-bdd3-a43df363b408 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:34.019499279 +0000 UTC m=+2.988031041 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/75f7d1d7-88e4-41e0-bdd3-a43df363b408-service-ca-bundle") pod "router-default-896ccfb6c-q4rg5" (UID: "75f7d1d7-88e4-41e0-bdd3-a43df363b408") : configmap references non-existent config key: service-ca.crt Nov 02 22:52:33.019867 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:33.019551 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15789a0d-e030-45e0-a129-41de738ef767-cluster-monitoring-operator-tls podName:15789a0d-e030-45e0-a129-41de738ef767 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:34.019541492 +0000 UTC m=+2.988073257 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/15789a0d-e030-45e0-a129-41de738ef767-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-mbmf9" (UID: "15789a0d-e030-45e0-a129-41de738ef767") : secret "cluster-monitoring-operator-tls" not found Nov 02 22:52:33.050400 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:33.050370 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0237eeb1_8c85_475e_8580_6dec324de705.slice/crio-f8cd1c3bc352c6b10bad63b1002882bb9e2547956cbd2a0ca1159436285a3a2e WatchSource:0}: Error finding container f8cd1c3bc352c6b10bad63b1002882bb9e2547956cbd2a0ca1159436285a3a2e: Status 404 returned error can't find the container with id f8cd1c3bc352c6b10bad63b1002882bb9e2547956cbd2a0ca1159436285a3a2e Nov 02 22:52:33.063754 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:33.063724 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46eae5da_cf0b_4663_bdeb_f8220fa08c05.slice/crio-b09b34b59f5ba9a0c00d160adb7cf7a2de548573d0d18bf39fc7b0d1e24e67e5 WatchSource:0}: Error finding container b09b34b59f5ba9a0c00d160adb7cf7a2de548573d0d18bf39fc7b0d1e24e67e5: Status 404 returned error can't find the container with id b09b34b59f5ba9a0c00d160adb7cf7a2de548573d0d18bf39fc7b0d1e24e67e5 Nov 02 22:52:33.214037 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:33.214002 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 02 22:52:33.214490 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:33.214472 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/1.log" Nov 02 22:52:33.214534 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:33.214513 5076 generic.go:356] "Generic (PLEG): container finished" podID="9c4b6fbd-74cf-41c6-bbae-92748ac25e1e" containerID="62fc934cdc0daa892fc0739a76d243792e37ab60b1d5a4f2f1da47c2b926f2d9" exitCode=255 Nov 02 22:52:33.214628 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:33.214595 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-c2gxc" event={"ID":"9c4b6fbd-74cf-41c6-bbae-92748ac25e1e","Type":"ContainerDied","Data":"62fc934cdc0daa892fc0739a76d243792e37ab60b1d5a4f2f1da47c2b926f2d9"} Nov 02 22:52:33.214724 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:33.214708 5076 scope.go:122] "RemoveContainer" containerID="61bc91e4752e543b447d5adcc7109202c5002dfd3c1c92a91e17d4830f0014e5" Nov 02 22:52:33.215103 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:33.215078 5076 scope.go:122] "RemoveContainer" containerID="62fc934cdc0daa892fc0739a76d243792e37ab60b1d5a4f2f1da47c2b926f2d9" Nov 02 22:52:33.215324 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:33.215302 5076 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-c2gxc_openshift-console-operator(9c4b6fbd-74cf-41c6-bbae-92748ac25e1e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-c2gxc" podUID="9c4b6fbd-74cf-41c6-bbae-92748ac25e1e" Nov 02 22:52:33.216385 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:33.216347 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" event={"ID":"46eae5da-cf0b-4663-bdeb-f8220fa08c05","Type":"ContainerStarted","Data":"c462a03a6914a2369de0cff964427ecd1ba7ef8f0c1085402d6e68e66734ba09"} Nov 02 22:52:33.216482 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:33.216391 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" event={"ID":"46eae5da-cf0b-4663-bdeb-f8220fa08c05","Type":"ContainerStarted","Data":"b09b34b59f5ba9a0c00d160adb7cf7a2de548573d0d18bf39fc7b0d1e24e67e5"} Nov 02 22:52:33.217955 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:33.217930 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-777f69544d-d75n6" event={"ID":"0237eeb1-8c85-475e-8580-6dec324de705","Type":"ContainerStarted","Data":"22e3aa5b6126ab82576f8f963fd84c26599ce5c8be1bb36eb3a7b10915f2297f"} Nov 02 22:52:33.218040 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:33.217975 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-777f69544d-d75n6" event={"ID":"0237eeb1-8c85-475e-8580-6dec324de705","Type":"ContainerStarted","Data":"f8cd1c3bc352c6b10bad63b1002882bb9e2547956cbd2a0ca1159436285a3a2e"} Nov 02 22:52:33.568489 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:33.568458 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h9lck_ac4a7fda-863e-4fb1-a98c-9b325a8c0df7/serve-healthcheck-canary/0.log" Nov 02 22:52:33.769084 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:33.769044 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-8mxfd_0b6b45c3-5275-4877-a84b-15e602100c40/migrator/0.log" Nov 02 22:52:33.969731 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:33.969655 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-f62lt_50d13ae8-4ab5-4ca7-a11a-ad632da19669/migrator/0.log" Nov 02 22:52:34.028783 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:34.028751 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/105406c8-6800-4562-9651-962bee9f8d73-metrics-tls\") pod \"dns-default-pvc5f\" (UID: \"105406c8-6800-4562-9651-962bee9f8d73\") " pod="openshift-dns/dns-default-pvc5f" Nov 02 22:52:34.028979 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:34.028791 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-metrics-certs\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:34.028979 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:34.028908 5076 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 22:52:34.028979 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:34.028946 5076 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 22:52:34.028979 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:34.028966 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/105406c8-6800-4562-9651-962bee9f8d73-metrics-tls podName:105406c8-6800-4562-9651-962bee9f8d73 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:36.028951746 +0000 UTC m=+4.997483509 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/105406c8-6800-4562-9651-962bee9f8d73-metrics-tls") pod "dns-default-pvc5f" (UID: "105406c8-6800-4562-9651-962bee9f8d73") : secret "dns-default-metrics-tls" not found Nov 02 22:52:34.029112 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:34.028992 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-metrics-certs podName:75f7d1d7-88e4-41e0-bdd3-a43df363b408 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:36.028981295 +0000 UTC m=+4.997513070 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-metrics-certs") pod "router-default-896ccfb6c-q4rg5" (UID: "75f7d1d7-88e4-41e0-bdd3-a43df363b408") : secret "router-metrics-certs-default" not found Nov 02 22:52:34.029112 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:34.029012 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs\") pod \"network-metrics-daemon-9zqrz\" (UID: \"fc69f07f-4001-4951-aff8-d3d6671cc5d4\") " pod="openshift-multus/network-metrics-daemon-9zqrz" Nov 02 22:52:34.029112 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:34.029048 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/15789a0d-e030-45e0-a129-41de738ef767-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mbmf9\" (UID: \"15789a0d-e030-45e0-a129-41de738ef767\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbmf9" Nov 02 22:52:34.029112 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:34.029086 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f7d1d7-88e4-41e0-bdd3-a43df363b408-service-ca-bundle\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:34.029112 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:34.029099 5076 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 22:52:34.029291 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:34.029126 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6c7074b-4caa-43fb-9fca-14ebb5abf958-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kkzvx\" (UID: \"b6c7074b-4caa-43fb-9fca-14ebb5abf958\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkzvx" Nov 02 22:52:34.029291 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:34.029139 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs podName:fc69f07f-4001-4951-aff8-d3d6671cc5d4 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:36.029127688 +0000 UTC m=+4.997659463 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs") pod "network-metrics-daemon-9zqrz" (UID: "fc69f07f-4001-4951-aff8-d3d6671cc5d4") : secret "metrics-daemon-secret" not found Nov 02 22:52:34.029291 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:34.029173 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/00b95c29-7df8-4602-bd0b-06b6281d5dc1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-cgpkk\" (UID: \"00b95c29-7df8-4602-bd0b-06b6281d5dc1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cgpkk" Nov 02 22:52:34.029291 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:34.029198 5076 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 22:52:34.029291 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:34.029223 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f98166f-8051-4dbd-9856-db692932050b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sv7kh\" (UID: \"9f98166f-8051-4dbd-9856-db692932050b\") " pod="openshift-insights/insights-runtime-extractor-sv7kh" Nov 02 22:52:34.029291 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:34.029251 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75f7d1d7-88e4-41e0-bdd3-a43df363b408-service-ca-bundle podName:75f7d1d7-88e4-41e0-bdd3-a43df363b408 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:36.029220543 +0000 UTC m=+4.997752326 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/75f7d1d7-88e4-41e0-bdd3-a43df363b408-service-ca-bundle") pod "router-default-896ccfb6c-q4rg5" (UID: "75f7d1d7-88e4-41e0-bdd3-a43df363b408") : configmap references non-existent config key: service-ca.crt Nov 02 22:52:34.029291 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:34.029268 5076 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 22:52:34.029291 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:34.029284 5076 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 22:52:34.029291 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:34.029293 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6c7074b-4caa-43fb-9fca-14ebb5abf958-networking-console-plugin-cert podName:b6c7074b-4caa-43fb-9fca-14ebb5abf958 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:36.029282619 +0000 UTC m=+4.997814385 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b6c7074b-4caa-43fb-9fca-14ebb5abf958-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kkzvx" (UID: "b6c7074b-4caa-43fb-9fca-14ebb5abf958") : secret "networking-console-plugin-cert" not found Nov 02 22:52:34.029560 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:34.029320 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00b95c29-7df8-4602-bd0b-06b6281d5dc1-samples-operator-tls podName:00b95c29-7df8-4602-bd0b-06b6281d5dc1 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:36.029309794 +0000 UTC m=+4.997841559 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/00b95c29-7df8-4602-bd0b-06b6281d5dc1-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-cgpkk" (UID: "00b95c29-7df8-4602-bd0b-06b6281d5dc1") : secret "samples-operator-tls" not found Nov 02 22:52:34.029560 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:34.029337 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f98166f-8051-4dbd-9856-db692932050b-insights-runtime-extractor-tls podName:9f98166f-8051-4dbd-9856-db692932050b nodeName:}" failed. No retries permitted until 2026-11-02 22:52:36.02932878 +0000 UTC m=+4.997860546 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f98166f-8051-4dbd-9856-db692932050b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sv7kh" (UID: "9f98166f-8051-4dbd-9856-db692932050b") : secret "insights-runtime-extractor-tls" not found Nov 02 22:52:34.031499 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:34.031475 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/15789a0d-e030-45e0-a129-41de738ef767-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mbmf9\" (UID: \"15789a0d-e030-45e0-a129-41de738ef767\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbmf9" Nov 02 22:52:34.049529 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:34.049503 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbmf9" Nov 02 22:52:34.169397 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:34.169367 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-ht25s_df418d6f-b9d3-4019-967c-9511be123e1f/kube-storage-version-migrator-operator/0.log" Nov 02 22:52:34.223725 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:34.223656 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 02 22:52:34.224085 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:34.223817 5076 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 22:52:34.224085 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:34.223827 5076 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 22:52:34.407635 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:34.407604 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-mbmf9"] Nov 02 22:52:34.410636 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:34.410609 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15789a0d_e030_45e0_a129_41de738ef767.slice/crio-6d4f19561b5ea3a288dbab871905d452fcdf17e50eaf12f364f77421aadceef0 WatchSource:0}: Error finding container 6d4f19561b5ea3a288dbab871905d452fcdf17e50eaf12f364f77421aadceef0: Status 404 returned error can't find the container with id 6d4f19561b5ea3a288dbab871905d452fcdf17e50eaf12f364f77421aadceef0 Nov 02 22:52:34.412439 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:34.412419 5076 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:52:35.228771 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:35.228738 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbmf9" event={"ID":"15789a0d-e030-45e0-a129-41de738ef767","Type":"ContainerStarted","Data":"6d4f19561b5ea3a288dbab871905d452fcdf17e50eaf12f364f77421aadceef0"} Nov 02 22:52:36.048121 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:36.048079 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f7d1d7-88e4-41e0-bdd3-a43df363b408-service-ca-bundle\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:36.048307 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:36.048133 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6c7074b-4caa-43fb-9fca-14ebb5abf958-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kkzvx\" (UID: \"b6c7074b-4caa-43fb-9fca-14ebb5abf958\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkzvx" Nov 02 22:52:36.048307 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:36.048162 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/00b95c29-7df8-4602-bd0b-06b6281d5dc1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-cgpkk\" (UID: \"00b95c29-7df8-4602-bd0b-06b6281d5dc1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cgpkk" Nov 02 22:52:36.048307 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:36.048196 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f98166f-8051-4dbd-9856-db692932050b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sv7kh\" (UID: \"9f98166f-8051-4dbd-9856-db692932050b\") " pod="openshift-insights/insights-runtime-extractor-sv7kh" Nov 02 22:52:36.048307 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:36.048243 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/105406c8-6800-4562-9651-962bee9f8d73-metrics-tls\") pod \"dns-default-pvc5f\" (UID: \"105406c8-6800-4562-9651-962bee9f8d73\") " pod="openshift-dns/dns-default-pvc5f" Nov 02 22:52:36.048307 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:36.048263 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-metrics-certs\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:36.048307 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:36.048266 5076 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 22:52:36.048307 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:36.048297 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs\") pod \"network-metrics-daemon-9zqrz\" (UID: \"fc69f07f-4001-4951-aff8-d3d6671cc5d4\") " pod="openshift-multus/network-metrics-daemon-9zqrz" Nov 02 22:52:36.048643 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:36.048349 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6c7074b-4caa-43fb-9fca-14ebb5abf958-networking-console-plugin-cert podName:b6c7074b-4caa-43fb-9fca-14ebb5abf958 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:40.048325375 +0000 UTC m=+9.016857154 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b6c7074b-4caa-43fb-9fca-14ebb5abf958-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kkzvx" (UID: "b6c7074b-4caa-43fb-9fca-14ebb5abf958") : secret "networking-console-plugin-cert" not found Nov 02 22:52:36.048643 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:36.048390 5076 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 22:52:36.048643 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:36.048403 5076 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 22:52:36.048643 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:36.048443 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f98166f-8051-4dbd-9856-db692932050b-insights-runtime-extractor-tls podName:9f98166f-8051-4dbd-9856-db692932050b nodeName:}" failed. No retries permitted until 2026-11-02 22:52:40.048430731 +0000 UTC m=+9.016962507 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f98166f-8051-4dbd-9856-db692932050b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sv7kh" (UID: "9f98166f-8051-4dbd-9856-db692932050b") : secret "insights-runtime-extractor-tls" not found Nov 02 22:52:36.048643 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:36.048450 5076 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 22:52:36.048643 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:36.048460 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/105406c8-6800-4562-9651-962bee9f8d73-metrics-tls podName:105406c8-6800-4562-9651-962bee9f8d73 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:40.04845223 +0000 UTC m=+9.016983994 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/105406c8-6800-4562-9651-962bee9f8d73-metrics-tls") pod "dns-default-pvc5f" (UID: "105406c8-6800-4562-9651-962bee9f8d73") : secret "dns-default-metrics-tls" not found Nov 02 22:52:36.048643 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:36.048474 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75f7d1d7-88e4-41e0-bdd3-a43df363b408-service-ca-bundle podName:75f7d1d7-88e4-41e0-bdd3-a43df363b408 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:40.048467058 +0000 UTC m=+9.016998845 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/75f7d1d7-88e4-41e0-bdd3-a43df363b408-service-ca-bundle") pod "router-default-896ccfb6c-q4rg5" (UID: "75f7d1d7-88e4-41e0-bdd3-a43df363b408") : configmap references non-existent config key: service-ca.crt Nov 02 22:52:36.048643 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:36.048396 5076 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 22:52:36.048643 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:36.048492 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-metrics-certs podName:75f7d1d7-88e4-41e0-bdd3-a43df363b408 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:40.048479139 +0000 UTC m=+9.017010915 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-metrics-certs") pod "router-default-896ccfb6c-q4rg5" (UID: "75f7d1d7-88e4-41e0-bdd3-a43df363b408") : secret "router-metrics-certs-default" not found Nov 02 22:52:36.048643 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:36.048510 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs podName:fc69f07f-4001-4951-aff8-d3d6671cc5d4 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:40.048501682 +0000 UTC m=+9.017033452 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs") pod "network-metrics-daemon-9zqrz" (UID: "fc69f07f-4001-4951-aff8-d3d6671cc5d4") : secret "metrics-daemon-secret" not found Nov 02 22:52:36.051143 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:36.051113 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/00b95c29-7df8-4602-bd0b-06b6281d5dc1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-cgpkk\" (UID: \"00b95c29-7df8-4602-bd0b-06b6281d5dc1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cgpkk" Nov 02 22:52:36.149563 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:36.149523 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cgpkk" Nov 02 22:52:36.183157 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:36.183129 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-c2gxc" Nov 02 22:52:36.183553 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:36.183532 5076 scope.go:122] "RemoveContainer" containerID="62fc934cdc0daa892fc0739a76d243792e37ab60b1d5a4f2f1da47c2b926f2d9" Nov 02 22:52:36.183766 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:36.183744 5076 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-c2gxc_openshift-console-operator(9c4b6fbd-74cf-41c6-bbae-92748ac25e1e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-c2gxc" podUID="9c4b6fbd-74cf-41c6-bbae-92748ac25e1e" Nov 02 22:52:36.361329 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:36.361246 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:36.532323 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:36.532123 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cgpkk"] Nov 02 22:52:37.239518 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:37.239480 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cgpkk" event={"ID":"00b95c29-7df8-4602-bd0b-06b6281d5dc1","Type":"ContainerStarted","Data":"421dca349b0b61b91aaf9ec011bc07bfc1f2420f616efcfbce3135b473e77208"} Nov 02 22:52:37.240879 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:37.240855 5076 generic.go:356] "Generic (PLEG): container finished" podID="df418d6f-b9d3-4019-967c-9511be123e1f" containerID="fcd8f266982c92e7a3d3a16cef84c06c4ad5a3efbf549823e26e26e1c034495c" exitCode=0 Nov 02 22:52:37.241022 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:37.240907 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ht25s" event={"ID":"df418d6f-b9d3-4019-967c-9511be123e1f","Type":"ContainerDied","Data":"fcd8f266982c92e7a3d3a16cef84c06c4ad5a3efbf549823e26e26e1c034495c"} Nov 02 22:52:37.241289 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:37.241273 5076 scope.go:122] "RemoveContainer" containerID="fcd8f266982c92e7a3d3a16cef84c06c4ad5a3efbf549823e26e26e1c034495c" Nov 02 22:52:37.324989 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:37.324956 5076 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-c2gxc" Nov 02 22:52:37.325396 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:37.325370 5076 scope.go:122] "RemoveContainer" containerID="62fc934cdc0daa892fc0739a76d243792e37ab60b1d5a4f2f1da47c2b926f2d9" Nov 02 22:52:37.325586 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:37.325564 5076 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-c2gxc_openshift-console-operator(9c4b6fbd-74cf-41c6-bbae-92748ac25e1e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-c2gxc" podUID="9c4b6fbd-74cf-41c6-bbae-92748ac25e1e" Nov 02 22:52:37.589820 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:37.589748 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:52:38.247537 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:38.246879 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ht25s" event={"ID":"df418d6f-b9d3-4019-967c-9511be123e1f","Type":"ContainerStarted","Data":"69e20934910383ae52bf6fbd78958d7264f855cc60282d63d73a90472283f9c2"} Nov 02 22:52:38.249013 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:38.248856 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbmf9" event={"ID":"15789a0d-e030-45e0-a129-41de738ef767","Type":"ContainerStarted","Data":"8b69a108fe31e4ee7f050b997d43fea2b984f8ac4f7659eae8a825e8a0781e94"} Nov 02 22:52:40.088014 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:40.087959 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f7d1d7-88e4-41e0-bdd3-a43df363b408-service-ca-bundle\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:40.088014 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:40.088023 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6c7074b-4caa-43fb-9fca-14ebb5abf958-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kkzvx\" (UID: \"b6c7074b-4caa-43fb-9fca-14ebb5abf958\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkzvx" Nov 02 22:52:40.088623 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:40.088059 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f98166f-8051-4dbd-9856-db692932050b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sv7kh\" (UID: \"9f98166f-8051-4dbd-9856-db692932050b\") " pod="openshift-insights/insights-runtime-extractor-sv7kh" Nov 02 22:52:40.088623 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:40.088095 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/105406c8-6800-4562-9651-962bee9f8d73-metrics-tls\") pod \"dns-default-pvc5f\" (UID: \"105406c8-6800-4562-9651-962bee9f8d73\") " pod="openshift-dns/dns-default-pvc5f" Nov 02 22:52:40.088623 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:40.088112 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-metrics-certs\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:40.088623 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:40.088156 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75f7d1d7-88e4-41e0-bdd3-a43df363b408-service-ca-bundle podName:75f7d1d7-88e4-41e0-bdd3-a43df363b408 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:48.088132168 +0000 UTC m=+17.056663949 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/75f7d1d7-88e4-41e0-bdd3-a43df363b408-service-ca-bundle") pod "router-default-896ccfb6c-q4rg5" (UID: "75f7d1d7-88e4-41e0-bdd3-a43df363b408") : configmap references non-existent config key: service-ca.crt Nov 02 22:52:40.088623 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:40.088191 5076 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 22:52:40.088623 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:40.088217 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs\") pod \"network-metrics-daemon-9zqrz\" (UID: \"fc69f07f-4001-4951-aff8-d3d6671cc5d4\") " pod="openshift-multus/network-metrics-daemon-9zqrz" Nov 02 22:52:40.088623 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:40.088242 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f98166f-8051-4dbd-9856-db692932050b-insights-runtime-extractor-tls podName:9f98166f-8051-4dbd-9856-db692932050b nodeName:}" failed. No retries permitted until 2026-11-02 22:52:48.088225158 +0000 UTC m=+17.056756930 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f98166f-8051-4dbd-9856-db692932050b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sv7kh" (UID: "9f98166f-8051-4dbd-9856-db692932050b") : secret "insights-runtime-extractor-tls" not found Nov 02 22:52:40.088623 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:40.088278 5076 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 22:52:40.088623 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:40.088328 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-metrics-certs podName:75f7d1d7-88e4-41e0-bdd3-a43df363b408 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:48.088314089 +0000 UTC m=+17.056845862 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-metrics-certs") pod "router-default-896ccfb6c-q4rg5" (UID: "75f7d1d7-88e4-41e0-bdd3-a43df363b408") : secret "router-metrics-certs-default" not found Nov 02 22:52:40.088623 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:40.088279 5076 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 22:52:40.088623 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:52:40.088367 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs podName:fc69f07f-4001-4951-aff8-d3d6671cc5d4 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:48.088357732 +0000 UTC m=+17.056889500 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs") pod "network-metrics-daemon-9zqrz" (UID: "fc69f07f-4001-4951-aff8-d3d6671cc5d4") : secret "metrics-daemon-secret" not found Nov 02 22:52:40.091026 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:40.091003 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/105406c8-6800-4562-9651-962bee9f8d73-metrics-tls\") pod \"dns-default-pvc5f\" (UID: \"105406c8-6800-4562-9651-962bee9f8d73\") " pod="openshift-dns/dns-default-pvc5f" Nov 02 22:52:40.091130 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:40.091109 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6c7074b-4caa-43fb-9fca-14ebb5abf958-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kkzvx\" (UID: \"b6c7074b-4caa-43fb-9fca-14ebb5abf958\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkzvx" Nov 02 22:52:40.257169 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:40.257133 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cgpkk" event={"ID":"00b95c29-7df8-4602-bd0b-06b6281d5dc1","Type":"ContainerStarted","Data":"7c6e6b09cdfc1a5eaf03c2e50b0c6aaa5a620caef90fa1445b87b847cd176f6c"} Nov 02 22:52:40.257338 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:40.257216 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-cgpkk" event={"ID":"00b95c29-7df8-4602-bd0b-06b6281d5dc1","Type":"ContainerStarted","Data":"a90fd25f0c6ff502dc360421188c3bb8ceec5a442babcfa6fc20f1f34f445ffb"} Nov 02 22:52:40.352575 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:40.352490 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-kkzvx" Nov 02 22:52:40.359867 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:40.359842 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pvc5f" Nov 02 22:52:40.724914 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:40.724847 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-kkzvx"] Nov 02 22:52:40.764237 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:40.764204 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6c7074b_4caa_43fb_9fca_14ebb5abf958.slice/crio-fab259ae1c20deeb55cf9acdaa51b16e2397a4a61204eb7b8215bd969733a715 WatchSource:0}: Error finding container fab259ae1c20deeb55cf9acdaa51b16e2397a4a61204eb7b8215bd969733a715: Status 404 returned error can't find the container with id fab259ae1c20deeb55cf9acdaa51b16e2397a4a61204eb7b8215bd969733a715 Nov 02 22:52:40.772512 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:40.772481 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pvc5f"] Nov 02 22:52:40.775311 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:40.775285 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod105406c8_6800_4562_9651_962bee9f8d73.slice/crio-ec694efc2ea5cd90909aa2ff58b4f4f61ce60cbeda2343305e3d7d79ddc32eb9 WatchSource:0}: Error finding container ec694efc2ea5cd90909aa2ff58b4f4f61ce60cbeda2343305e3d7d79ddc32eb9: Status 404 returned error can't find the container with id ec694efc2ea5cd90909aa2ff58b4f4f61ce60cbeda2343305e3d7d79ddc32eb9 Nov 02 22:52:41.020115 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:41.020028 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rmgpt" Nov 02 22:52:41.261305 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:41.261265 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pvc5f" event={"ID":"105406c8-6800-4562-9651-962bee9f8d73","Type":"ContainerStarted","Data":"ec694efc2ea5cd90909aa2ff58b4f4f61ce60cbeda2343305e3d7d79ddc32eb9"} Nov 02 22:52:41.262471 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:41.262437 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-kkzvx" event={"ID":"b6c7074b-4caa-43fb-9fca-14ebb5abf958","Type":"ContainerStarted","Data":"fab259ae1c20deeb55cf9acdaa51b16e2397a4a61204eb7b8215bd969733a715"} Nov 02 22:52:41.929153 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:41.929126 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:41.929404 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:41.929390 5076 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 22:52:41.954542 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:41.954500 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:52:43.271935 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:43.271883 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-kkzvx" event={"ID":"b6c7074b-4caa-43fb-9fca-14ebb5abf958","Type":"ContainerStarted","Data":"605ef130807ec0c208322bf9c9d29556ae65409da1821237bef1ab4d215b3fe7"} Nov 02 22:52:44.277103 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:44.277018 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pvc5f" event={"ID":"105406c8-6800-4562-9651-962bee9f8d73","Type":"ContainerStarted","Data":"8ff86e4f8e0918d8346f6c0d8b3757d0188f7f56713ab305b566de7957e943f1"} Nov 02 22:52:44.277103 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:44.277072 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pvc5f" event={"ID":"105406c8-6800-4562-9651-962bee9f8d73","Type":"ContainerStarted","Data":"6321d5d9308e2efd629ed84d2aed87147f86aa37161fee58c1fd821a9488d26a"} Nov 02 22:52:44.277550 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:44.277303 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pvc5f" Nov 02 22:52:45.140406 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:45.140368 5076 scope.go:122] "RemoveContainer" containerID="62fc934cdc0daa892fc0739a76d243792e37ab60b1d5a4f2f1da47c2b926f2d9" Nov 02 22:52:45.282065 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:45.282038 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 02 22:52:45.282467 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:45.282112 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-c2gxc" event={"ID":"9c4b6fbd-74cf-41c6-bbae-92748ac25e1e","Type":"ContainerStarted","Data":"9dd5ae9df24e5bd4fae09be5fa477cc839b367bdfce3d4d9cf280311c845cff8"} Nov 02 22:52:46.521934 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:46.521904 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bp6bf" Nov 02 22:52:48.165240 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.165201 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f98166f-8051-4dbd-9856-db692932050b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sv7kh\" (UID: \"9f98166f-8051-4dbd-9856-db692932050b\") " pod="openshift-insights/insights-runtime-extractor-sv7kh" Nov 02 22:52:48.165638 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.165271 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-metrics-certs\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:48.165638 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.165318 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs\") pod \"network-metrics-daemon-9zqrz\" (UID: \"fc69f07f-4001-4951-aff8-d3d6671cc5d4\") " pod="openshift-multus/network-metrics-daemon-9zqrz" Nov 02 22:52:48.165638 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.165451 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f7d1d7-88e4-41e0-bdd3-a43df363b408-service-ca-bundle\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:48.166138 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.166111 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f7d1d7-88e4-41e0-bdd3-a43df363b408-service-ca-bundle\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:48.167703 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.167673 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc69f07f-4001-4951-aff8-d3d6671cc5d4-metrics-certs\") pod \"network-metrics-daemon-9zqrz\" (UID: \"fc69f07f-4001-4951-aff8-d3d6671cc5d4\") " pod="openshift-multus/network-metrics-daemon-9zqrz" Nov 02 22:52:48.167803 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.167676 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f98166f-8051-4dbd-9856-db692932050b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sv7kh\" (UID: \"9f98166f-8051-4dbd-9856-db692932050b\") " pod="openshift-insights/insights-runtime-extractor-sv7kh" Nov 02 22:52:48.167872 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.167855 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75f7d1d7-88e4-41e0-bdd3-a43df363b408-metrics-certs\") pod \"router-default-896ccfb6c-q4rg5\" (UID: \"75f7d1d7-88e4-41e0-bdd3-a43df363b408\") " pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:48.448818 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.448777 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:48.453940 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.453918 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zqrz" Nov 02 22:52:48.461089 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.461070 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sv7kh" Nov 02 22:52:48.652000 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.651967 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wtk4j"] Nov 02 22:52:48.678086 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.678058 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wtk4j" Nov 02 22:52:48.699714 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.699643 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-777f69544d-d75n6"] Nov 02 22:52:48.705332 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.705295 5076 patch_prober.go:29] interesting pod/image-registry-777f69544d-d75n6 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 02 22:52:48.705485 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.705355 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-777f69544d-d75n6" podUID="0237eeb1-8c85-475e-8580-6dec324de705" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:52:48.731822 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.731792 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 02 22:52:48.732162 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.731991 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-tgh5g\"" Nov 02 22:52:48.747355 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.747326 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-p44bf"] Nov 02 22:52:48.764419 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.764394 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-p44bf" Nov 02 22:52:48.770930 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.770880 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/be9397ed-f3c5-4b5d-a300-04f04f8fd355-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-wtk4j\" (UID: \"be9397ed-f3c5-4b5d-a300-04f04f8fd355\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wtk4j" Nov 02 22:52:48.780152 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.780127 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6fffcf7b58-2vvth"] Nov 02 22:52:48.796299 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.796273 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fffcf7b58-2vvth" Nov 02 22:52:48.806178 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.806153 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6fffcf7b58-2vvth"] Nov 02 22:52:48.814371 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.814347 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 02 22:52:48.814371 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.814349 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 02 22:52:48.831623 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.831596 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-5hjn2\"" Nov 02 22:52:48.852851 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.852829 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-896ccfb6c-q4rg5"] Nov 02 22:52:48.855541 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:48.855514 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f7d1d7_88e4_41e0_bdd3_a43df363b408.slice/crio-91cc351c125131329fdb631b591f0697b2456abc52ff64f52039573c6f990c92 WatchSource:0}: Error finding container 91cc351c125131329fdb631b591f0697b2456abc52ff64f52039573c6f990c92: Status 404 returned error can't find the container with id 91cc351c125131329fdb631b591f0697b2456abc52ff64f52039573c6f990c92 Nov 02 22:52:48.866881 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.866851 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wtk4j"] Nov 02 22:52:48.872433 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.872403 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e5318a9-b5ae-46bd-bd99-de3d03b0771f-bound-sa-token\") pod \"image-registry-6fffcf7b58-2vvth\" (UID: \"5e5318a9-b5ae-46bd-bd99-de3d03b0771f\") " pod="openshift-image-registry/image-registry-6fffcf7b58-2vvth" Nov 02 22:52:48.872539 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.872453 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e5318a9-b5ae-46bd-bd99-de3d03b0771f-registry-tls\") pod \"image-registry-6fffcf7b58-2vvth\" (UID: \"5e5318a9-b5ae-46bd-bd99-de3d03b0771f\") " pod="openshift-image-registry/image-registry-6fffcf7b58-2vvth" Nov 02 22:52:48.872605 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.872528 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5r6f\" (UniqueName: \"kubernetes.io/projected/5e5318a9-b5ae-46bd-bd99-de3d03b0771f-kube-api-access-f5r6f\") pod \"image-registry-6fffcf7b58-2vvth\" (UID: \"5e5318a9-b5ae-46bd-bd99-de3d03b0771f\") " pod="openshift-image-registry/image-registry-6fffcf7b58-2vvth" Nov 02 22:52:48.872663 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.872606 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e1fcec6-163f-4442-8444-19c2c00986f0-tmp\") pod \"downloads-6dd789c8b7-p44bf\" (UID: \"4e1fcec6-163f-4442-8444-19c2c00986f0\") " pod="openshift-console/downloads-6dd789c8b7-p44bf" Nov 02 22:52:48.872663 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.872643 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5e5318a9-b5ae-46bd-bd99-de3d03b0771f-installation-pull-secrets\") pod \"image-registry-6fffcf7b58-2vvth\" (UID: \"5e5318a9-b5ae-46bd-bd99-de3d03b0771f\") " pod="openshift-image-registry/image-registry-6fffcf7b58-2vvth" Nov 02 22:52:48.872731 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.872692 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr7c8\" (UniqueName: \"kubernetes.io/projected/4e1fcec6-163f-4442-8444-19c2c00986f0-kube-api-access-nr7c8\") pod \"downloads-6dd789c8b7-p44bf\" (UID: \"4e1fcec6-163f-4442-8444-19c2c00986f0\") " pod="openshift-console/downloads-6dd789c8b7-p44bf" Nov 02 22:52:48.872731 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.872717 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5e5318a9-b5ae-46bd-bd99-de3d03b0771f-registry-certificates\") pod \"image-registry-6fffcf7b58-2vvth\" (UID: \"5e5318a9-b5ae-46bd-bd99-de3d03b0771f\") " pod="openshift-image-registry/image-registry-6fffcf7b58-2vvth" Nov 02 22:52:48.872799 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.872751 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5e5318a9-b5ae-46bd-bd99-de3d03b0771f-ca-trust-extracted\") pod \"image-registry-6fffcf7b58-2vvth\" (UID: \"5e5318a9-b5ae-46bd-bd99-de3d03b0771f\") " pod="openshift-image-registry/image-registry-6fffcf7b58-2vvth" Nov 02 22:52:48.872840 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.872796 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e5318a9-b5ae-46bd-bd99-de3d03b0771f-trusted-ca\") pod \"image-registry-6fffcf7b58-2vvth\" (UID: \"5e5318a9-b5ae-46bd-bd99-de3d03b0771f\") " pod="openshift-image-registry/image-registry-6fffcf7b58-2vvth" Nov 02 22:52:48.872933 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.872844 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/be9397ed-f3c5-4b5d-a300-04f04f8fd355-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-wtk4j\" (UID: \"be9397ed-f3c5-4b5d-a300-04f04f8fd355\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wtk4j" Nov 02 22:52:48.872933 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.872870 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5e5318a9-b5ae-46bd-bd99-de3d03b0771f-image-registry-private-configuration\") pod \"image-registry-6fffcf7b58-2vvth\" (UID: \"5e5318a9-b5ae-46bd-bd99-de3d03b0771f\") " pod="openshift-image-registry/image-registry-6fffcf7b58-2vvth" Nov 02 22:52:48.875614 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.875592 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/be9397ed-f3c5-4b5d-a300-04f04f8fd355-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-wtk4j\" (UID: \"be9397ed-f3c5-4b5d-a300-04f04f8fd355\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wtk4j" Nov 02 22:52:48.892841 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.892816 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9zqrz"] Nov 02 22:52:48.896720 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:48.896691 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc69f07f_4001_4951_aff8_d3d6671cc5d4.slice/crio-5c6a3d7b4e4cfecb87611b46f6167957ceb8e6009206503b248f4667555c19e6 WatchSource:0}: Error finding container 5c6a3d7b4e4cfecb87611b46f6167957ceb8e6009206503b248f4667555c19e6: Status 404 returned error can't find the container with id 5c6a3d7b4e4cfecb87611b46f6167957ceb8e6009206503b248f4667555c19e6 Nov 02 22:52:48.949519 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.949483 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sv7kh"] Nov 02 22:52:48.953486 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:48.953458 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f98166f_8051_4dbd_9856_db692932050b.slice/crio-ce596bd1f950e957505d60b0f6203e607e5e4bd54b8688e391cc9461d1d7841f WatchSource:0}: Error finding container ce596bd1f950e957505d60b0f6203e607e5e4bd54b8688e391cc9461d1d7841f: Status 404 returned error can't find the container with id ce596bd1f950e957505d60b0f6203e607e5e4bd54b8688e391cc9461d1d7841f Nov 02 22:52:48.973257 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.973230 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5e5318a9-b5ae-46bd-bd99-de3d03b0771f-installation-pull-secrets\") pod \"image-registry-6fffcf7b58-2vvth\" (UID: \"5e5318a9-b5ae-46bd-bd99-de3d03b0771f\") " pod="openshift-image-registry/image-registry-6fffcf7b58-2vvth" Nov 02 22:52:48.973385 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.973287 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr7c8\" (UniqueName: \"kubernetes.io/projected/4e1fcec6-163f-4442-8444-19c2c00986f0-kube-api-access-nr7c8\") pod \"downloads-6dd789c8b7-p44bf\" (UID: \"4e1fcec6-163f-4442-8444-19c2c00986f0\") " pod="openshift-console/downloads-6dd789c8b7-p44bf" Nov 02 22:52:48.973385 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.973313 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5e5318a9-b5ae-46bd-bd99-de3d03b0771f-registry-certificates\") pod \"image-registry-6fffcf7b58-2vvth\" (UID: \"5e5318a9-b5ae-46bd-bd99-de3d03b0771f\") " pod="openshift-image-registry/image-registry-6fffcf7b58-2vvth" Nov 02 22:52:48.973385 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.973344 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5e5318a9-b5ae-46bd-bd99-de3d03b0771f-ca-trust-extracted\") pod \"image-registry-6fffcf7b58-2vvth\" (UID: \"5e5318a9-b5ae-46bd-bd99-de3d03b0771f\") " pod="openshift-image-registry/image-registry-6fffcf7b58-2vvth" Nov 02 22:52:48.973385 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.973381 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e5318a9-b5ae-46bd-bd99-de3d03b0771f-trusted-ca\") pod \"image-registry-6fffcf7b58-2vvth\" (UID: \"5e5318a9-b5ae-46bd-bd99-de3d03b0771f\") " pod="openshift-image-registry/image-registry-6fffcf7b58-2vvth" Nov 02 22:52:48.973589 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.973420 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5e5318a9-b5ae-46bd-bd99-de3d03b0771f-image-registry-private-configuration\") pod \"image-registry-6fffcf7b58-2vvth\" (UID: \"5e5318a9-b5ae-46bd-bd99-de3d03b0771f\") " pod="openshift-image-registry/image-registry-6fffcf7b58-2vvth" Nov 02 22:52:48.973589 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.973465 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e5318a9-b5ae-46bd-bd99-de3d03b0771f-bound-sa-token\") pod \"image-registry-6fffcf7b58-2vvth\" (UID: \"5e5318a9-b5ae-46bd-bd99-de3d03b0771f\") " pod="openshift-image-registry/image-registry-6fffcf7b58-2vvth" Nov 02 22:52:48.973589 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.973497 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e5318a9-b5ae-46bd-bd99-de3d03b0771f-registry-tls\") pod \"image-registry-6fffcf7b58-2vvth\" (UID: \"5e5318a9-b5ae-46bd-bd99-de3d03b0771f\") " pod="openshift-image-registry/image-registry-6fffcf7b58-2vvth" Nov 02 22:52:48.973589 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.973525 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5r6f\" (UniqueName: \"kubernetes.io/projected/5e5318a9-b5ae-46bd-bd99-de3d03b0771f-kube-api-access-f5r6f\") pod \"image-registry-6fffcf7b58-2vvth\" (UID: \"5e5318a9-b5ae-46bd-bd99-de3d03b0771f\") " pod="openshift-image-registry/image-registry-6fffcf7b58-2vvth" Nov 02 22:52:48.973589 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.973558 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e1fcec6-163f-4442-8444-19c2c00986f0-tmp\") pod \"downloads-6dd789c8b7-p44bf\" (UID: \"4e1fcec6-163f-4442-8444-19c2c00986f0\") " pod="openshift-console/downloads-6dd789c8b7-p44bf" Nov 02 22:52:48.973939 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.973912 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5e5318a9-b5ae-46bd-bd99-de3d03b0771f-ca-trust-extracted\") pod \"image-registry-6fffcf7b58-2vvth\" (UID: \"5e5318a9-b5ae-46bd-bd99-de3d03b0771f\") " pod="openshift-image-registry/image-registry-6fffcf7b58-2vvth" Nov 02 22:52:48.974054 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.973979 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e1fcec6-163f-4442-8444-19c2c00986f0-tmp\") pod \"downloads-6dd789c8b7-p44bf\" (UID: \"4e1fcec6-163f-4442-8444-19c2c00986f0\") " pod="openshift-console/downloads-6dd789c8b7-p44bf" Nov 02 22:52:48.974245 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.974217 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5e5318a9-b5ae-46bd-bd99-de3d03b0771f-registry-certificates\") pod \"image-registry-6fffcf7b58-2vvth\" (UID: \"5e5318a9-b5ae-46bd-bd99-de3d03b0771f\") " pod="openshift-image-registry/image-registry-6fffcf7b58-2vvth" Nov 02 22:52:48.974365 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.974304 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e5318a9-b5ae-46bd-bd99-de3d03b0771f-trusted-ca\") pod \"image-registry-6fffcf7b58-2vvth\" (UID: \"5e5318a9-b5ae-46bd-bd99-de3d03b0771f\") " pod="openshift-image-registry/image-registry-6fffcf7b58-2vvth" Nov 02 22:52:48.975983 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.975952 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5e5318a9-b5ae-46bd-bd99-de3d03b0771f-image-registry-private-configuration\") pod \"image-registry-6fffcf7b58-2vvth\" (UID: \"5e5318a9-b5ae-46bd-bd99-de3d03b0771f\") " pod="openshift-image-registry/image-registry-6fffcf7b58-2vvth" Nov 02 22:52:48.976071 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.976032 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5e5318a9-b5ae-46bd-bd99-de3d03b0771f-installation-pull-secrets\") pod \"image-registry-6fffcf7b58-2vvth\" (UID: \"5e5318a9-b5ae-46bd-bd99-de3d03b0771f\") " pod="openshift-image-registry/image-registry-6fffcf7b58-2vvth" Nov 02 22:52:48.976146 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:48.976127 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e5318a9-b5ae-46bd-bd99-de3d03b0771f-registry-tls\") pod \"image-registry-6fffcf7b58-2vvth\" (UID: \"5e5318a9-b5ae-46bd-bd99-de3d03b0771f\") " pod="openshift-image-registry/image-registry-6fffcf7b58-2vvth" Nov 02 22:52:49.001202 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:49.001177 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wtk4j" Nov 02 22:52:49.007792 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:49.007763 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-p44bf"] Nov 02 22:52:49.028096 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:49.028060 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e5318a9-b5ae-46bd-bd99-de3d03b0771f-bound-sa-token\") pod \"image-registry-6fffcf7b58-2vvth\" (UID: \"5e5318a9-b5ae-46bd-bd99-de3d03b0771f\") " pod="openshift-image-registry/image-registry-6fffcf7b58-2vvth" Nov 02 22:52:49.028222 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:49.028182 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr7c8\" (UniqueName: \"kubernetes.io/projected/4e1fcec6-163f-4442-8444-19c2c00986f0-kube-api-access-nr7c8\") pod \"downloads-6dd789c8b7-p44bf\" (UID: \"4e1fcec6-163f-4442-8444-19c2c00986f0\") " pod="openshift-console/downloads-6dd789c8b7-p44bf" Nov 02 22:52:49.028534 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:49.028512 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5r6f\" (UniqueName: \"kubernetes.io/projected/5e5318a9-b5ae-46bd-bd99-de3d03b0771f-kube-api-access-f5r6f\") pod \"image-registry-6fffcf7b58-2vvth\" (UID: \"5e5318a9-b5ae-46bd-bd99-de3d03b0771f\") " pod="openshift-image-registry/image-registry-6fffcf7b58-2vvth" Nov 02 22:52:49.090835 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:49.090797 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-p44bf" Nov 02 22:52:49.120515 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:49.120488 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fffcf7b58-2vvth" Nov 02 22:52:49.288961 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:49.288930 5076 patch_prober.go:29] interesting pod/image-registry-7b5b8cd649-z4rbs container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 02 22:52:49.289319 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:49.288978 5076 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" podUID="46eae5da-cf0b-4663-bdeb-f8220fa08c05" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:52:49.297798 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:49.297765 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sv7kh" event={"ID":"9f98166f-8051-4dbd-9856-db692932050b","Type":"ContainerStarted","Data":"5207fdf0320d21ecc3209314b73ee9211d9d9633b8275df5bd5275c52d22bde9"} Nov 02 22:52:49.297953 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:49.297832 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sv7kh" event={"ID":"9f98166f-8051-4dbd-9856-db692932050b","Type":"ContainerStarted","Data":"ce596bd1f950e957505d60b0f6203e607e5e4bd54b8688e391cc9461d1d7841f"} Nov 02 22:52:49.299241 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:49.299205 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-896ccfb6c-q4rg5" event={"ID":"75f7d1d7-88e4-41e0-bdd3-a43df363b408","Type":"ContainerStarted","Data":"e24bec935966ce7cbc3ec5e51b1918c2f95aeea99c133d12f27e09a303cb13f9"} Nov 02 22:52:49.299363 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:49.299259 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-896ccfb6c-q4rg5" event={"ID":"75f7d1d7-88e4-41e0-bdd3-a43df363b408","Type":"ContainerStarted","Data":"91cc351c125131329fdb631b591f0697b2456abc52ff64f52039573c6f990c92"} Nov 02 22:52:49.300295 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:49.300268 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9zqrz" event={"ID":"fc69f07f-4001-4951-aff8-d3d6671cc5d4","Type":"ContainerStarted","Data":"5c6a3d7b4e4cfecb87611b46f6167957ceb8e6009206503b248f4667555c19e6"} Nov 02 22:52:49.387184 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:49.387158 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wtk4j"] Nov 02 22:52:49.389694 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:49.389652 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe9397ed_f3c5_4b5d_a300_04f04f8fd355.slice/crio-835b3443aa36567e9d050df66ac047201bcb42b8a0b9af3fdedd5c025e27ec82 WatchSource:0}: Error finding container 835b3443aa36567e9d050df66ac047201bcb42b8a0b9af3fdedd5c025e27ec82: Status 404 returned error can't find the container with id 835b3443aa36567e9d050df66ac047201bcb42b8a0b9af3fdedd5c025e27ec82 Nov 02 22:52:49.449512 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:49.449478 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:49.452093 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:49.452069 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:49.466210 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:49.466155 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-p44bf"] Nov 02 22:52:49.470173 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:49.470140 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e1fcec6_163f_4442_8444_19c2c00986f0.slice/crio-24542d4d4ed21e15ac97f2e6a5b28a4a20a88a55b012694fee7dee3526760a89 WatchSource:0}: Error finding container 24542d4d4ed21e15ac97f2e6a5b28a4a20a88a55b012694fee7dee3526760a89: Status 404 returned error can't find the container with id 24542d4d4ed21e15ac97f2e6a5b28a4a20a88a55b012694fee7dee3526760a89 Nov 02 22:52:49.520663 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:49.520635 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6fffcf7b58-2vvth"] Nov 02 22:52:49.523525 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:52:49.523457 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e5318a9_b5ae_46bd_bd99_de3d03b0771f.slice/crio-5384e5436e90469e1c01e69ebfb29ab5cbb522ecb467f5d5a875d6282709b530 WatchSource:0}: Error finding container 5384e5436e90469e1c01e69ebfb29ab5cbb522ecb467f5d5a875d6282709b530: Status 404 returned error can't find the container with id 5384e5436e90469e1c01e69ebfb29ab5cbb522ecb467f5d5a875d6282709b530 Nov 02 22:52:50.308504 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:50.308462 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wtk4j" event={"ID":"be9397ed-f3c5-4b5d-a300-04f04f8fd355","Type":"ContainerStarted","Data":"835b3443aa36567e9d050df66ac047201bcb42b8a0b9af3fdedd5c025e27ec82"} Nov 02 22:52:50.310450 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:50.310420 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fffcf7b58-2vvth" event={"ID":"5e5318a9-b5ae-46bd-bd99-de3d03b0771f","Type":"ContainerStarted","Data":"4fe85141a89b88bad0a3a9b625c94f1a102913d476cf126889a5315391b9d0e9"} Nov 02 22:52:50.310586 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:50.310473 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fffcf7b58-2vvth" event={"ID":"5e5318a9-b5ae-46bd-bd99-de3d03b0771f","Type":"ContainerStarted","Data":"5384e5436e90469e1c01e69ebfb29ab5cbb522ecb467f5d5a875d6282709b530"} Nov 02 22:52:50.310586 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:50.310513 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6fffcf7b58-2vvth" Nov 02 22:52:50.311968 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:50.311942 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-p44bf" event={"ID":"4e1fcec6-163f-4442-8444-19c2c00986f0","Type":"ContainerStarted","Data":"24542d4d4ed21e15ac97f2e6a5b28a4a20a88a55b012694fee7dee3526760a89"} Nov 02 22:52:50.312292 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:50.312272 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:50.313621 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:50.313602 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-896ccfb6c-q4rg5" Nov 02 22:52:50.424876 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:50.424814 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6fffcf7b58-2vvth" podStartSLOduration=2.424799363 podStartE2EDuration="2.424799363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:50.424281668 +0000 UTC m=+19.392813453" watchObservedRunningTime="2026-11-02 22:52:50.424799363 +0000 UTC m=+19.393331147" Nov 02 22:52:51.324099 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:51.324017 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9zqrz" event={"ID":"fc69f07f-4001-4951-aff8-d3d6671cc5d4","Type":"ContainerStarted","Data":"58becf1aec3f85c40b56c9e8f872851ebf07e68ecec67aff365d9e464908afda"} Nov 02 22:52:51.326799 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:51.326770 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sv7kh" event={"ID":"9f98166f-8051-4dbd-9856-db692932050b","Type":"ContainerStarted","Data":"d2d024babb3492f2c6aaf30eee280fcf99e70fd50a05f1907055f8235be1aece"} Nov 02 22:52:51.899615 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:51.899588 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zqssw" Nov 02 22:52:51.960174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:51.960131 5076 patch_prober.go:29] interesting pod/image-registry-7b5b8cd649-z4rbs container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 02 22:52:51.960356 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:51.960185 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" podUID="46eae5da-cf0b-4663-bdeb-f8220fa08c05" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:52:52.332762 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:52.332726 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wtk4j" event={"ID":"be9397ed-f3c5-4b5d-a300-04f04f8fd355","Type":"ContainerStarted","Data":"900f87c9c7e8da97d27c35dbf95ed2bb2a3247ee062f152b9e2332df93e397a7"} Nov 02 22:52:52.333690 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:52.333670 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wtk4j" Nov 02 22:52:52.336618 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:52.336053 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9zqrz" event={"ID":"fc69f07f-4001-4951-aff8-d3d6671cc5d4","Type":"ContainerStarted","Data":"9f005496748daeb1becdf1ee46eed18d5ecbbad28acfaa6477b389de766cdcb2"} Nov 02 22:52:52.339063 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:52.339036 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wtk4j" Nov 02 22:52:52.440685 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:52.440634 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wtk4j" podStartSLOduration=2.000832461 podStartE2EDuration="4.440618668s" totalImagesPullingTime="2.439786207s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:52:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:52.4399163 +0000 UTC m=+21.408448146" watchObservedRunningTime="2026-11-02 22:52:52.440618668 +0000 UTC m=+21.409150450" Nov 02 22:52:54.285795 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:54.285760 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pvc5f" Nov 02 22:52:54.348071 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:54.348034 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sv7kh" event={"ID":"9f98166f-8051-4dbd-9856-db692932050b","Type":"ContainerStarted","Data":"7cf1e44613e96e1b2b9b0ea4e8a9cb1c0e7dbe5b62256fd4260e1c33e37389d0"} Nov 02 22:52:55.282411 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:55.282376 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-c2gxc" Nov 02 22:52:55.288412 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:55.288217 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-c2gxc" Nov 02 22:52:58.705319 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:58.705280 5076 patch_prober.go:29] interesting pod/image-registry-777f69544d-d75n6 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 02 22:52:58.705859 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:58.705345 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-777f69544d-d75n6" podUID="0237eeb1-8c85-475e-8580-6dec324de705" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:52:59.289404 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:59.289365 5076 patch_prober.go:29] interesting pod/image-registry-7b5b8cd649-z4rbs container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 02 22:52:59.289589 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:52:59.289425 5076 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" podUID="46eae5da-cf0b-4663-bdeb-f8220fa08c05" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:53:00.774346 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:00.774306 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kf24r"] Nov 02 22:53:00.777768 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:00.777745 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kf24r" Nov 02 22:53:00.877421 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:00.877254 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 22:53:00.877421 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:00.877269 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 22:53:00.877421 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:00.877326 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rnj8n\"" Nov 02 22:53:00.877421 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:00.877394 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 22:53:00.877421 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:00.877419 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 22:53:00.883484 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:00.883451 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/14242834-56aa-458b-bacc-bb5a4aa97937-node-exporter-textfile\") pod \"node-exporter-kf24r\" (UID: \"14242834-56aa-458b-bacc-bb5a4aa97937\") " pod="openshift-monitoring/node-exporter-kf24r" Nov 02 22:53:00.883646 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:00.883496 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/14242834-56aa-458b-bacc-bb5a4aa97937-node-exporter-wtmp\") pod \"node-exporter-kf24r\" (UID: \"14242834-56aa-458b-bacc-bb5a4aa97937\") " pod="openshift-monitoring/node-exporter-kf24r" Nov 02 22:53:00.883646 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:00.883546 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14242834-56aa-458b-bacc-bb5a4aa97937-sys\") pod \"node-exporter-kf24r\" (UID: \"14242834-56aa-458b-bacc-bb5a4aa97937\") " pod="openshift-monitoring/node-exporter-kf24r" Nov 02 22:53:00.883646 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:00.883595 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14242834-56aa-458b-bacc-bb5a4aa97937-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kf24r\" (UID: \"14242834-56aa-458b-bacc-bb5a4aa97937\") " pod="openshift-monitoring/node-exporter-kf24r" Nov 02 22:53:00.883646 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:00.883635 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/14242834-56aa-458b-bacc-bb5a4aa97937-root\") pod \"node-exporter-kf24r\" (UID: \"14242834-56aa-458b-bacc-bb5a4aa97937\") " pod="openshift-monitoring/node-exporter-kf24r" Nov 02 22:53:00.883825 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:00.883677 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14242834-56aa-458b-bacc-bb5a4aa97937-metrics-client-ca\") pod \"node-exporter-kf24r\" (UID: \"14242834-56aa-458b-bacc-bb5a4aa97937\") " pod="openshift-monitoring/node-exporter-kf24r" Nov 02 22:53:00.883825 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:00.883717 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/14242834-56aa-458b-bacc-bb5a4aa97937-node-exporter-tls\") pod \"node-exporter-kf24r\" (UID: \"14242834-56aa-458b-bacc-bb5a4aa97937\") " pod="openshift-monitoring/node-exporter-kf24r" Nov 02 22:53:00.883825 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:00.883749 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/14242834-56aa-458b-bacc-bb5a4aa97937-node-exporter-accelerators-collector-config\") pod \"node-exporter-kf24r\" (UID: \"14242834-56aa-458b-bacc-bb5a4aa97937\") " pod="openshift-monitoring/node-exporter-kf24r" Nov 02 22:53:00.883825 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:00.883788 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7rz6\" (UniqueName: \"kubernetes.io/projected/14242834-56aa-458b-bacc-bb5a4aa97937-kube-api-access-z7rz6\") pod \"node-exporter-kf24r\" (UID: \"14242834-56aa-458b-bacc-bb5a4aa97937\") " pod="openshift-monitoring/node-exporter-kf24r" Nov 02 22:53:00.985092 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:00.985038 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14242834-56aa-458b-bacc-bb5a4aa97937-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kf24r\" (UID: \"14242834-56aa-458b-bacc-bb5a4aa97937\") " pod="openshift-monitoring/node-exporter-kf24r" Nov 02 22:53:00.985272 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:00.985106 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/14242834-56aa-458b-bacc-bb5a4aa97937-root\") pod \"node-exporter-kf24r\" (UID: \"14242834-56aa-458b-bacc-bb5a4aa97937\") " pod="openshift-monitoring/node-exporter-kf24r" Nov 02 22:53:00.985272 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:00.985139 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14242834-56aa-458b-bacc-bb5a4aa97937-metrics-client-ca\") pod \"node-exporter-kf24r\" (UID: \"14242834-56aa-458b-bacc-bb5a4aa97937\") " pod="openshift-monitoring/node-exporter-kf24r" Nov 02 22:53:00.985272 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:00.985166 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/14242834-56aa-458b-bacc-bb5a4aa97937-node-exporter-tls\") pod \"node-exporter-kf24r\" (UID: \"14242834-56aa-458b-bacc-bb5a4aa97937\") " pod="openshift-monitoring/node-exporter-kf24r" Nov 02 22:53:00.985272 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:00.985191 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/14242834-56aa-458b-bacc-bb5a4aa97937-node-exporter-accelerators-collector-config\") pod \"node-exporter-kf24r\" (UID: \"14242834-56aa-458b-bacc-bb5a4aa97937\") " pod="openshift-monitoring/node-exporter-kf24r" Nov 02 22:53:00.985272 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:00.985216 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7rz6\" (UniqueName: \"kubernetes.io/projected/14242834-56aa-458b-bacc-bb5a4aa97937-kube-api-access-z7rz6\") pod \"node-exporter-kf24r\" (UID: \"14242834-56aa-458b-bacc-bb5a4aa97937\") " pod="openshift-monitoring/node-exporter-kf24r" Nov 02 22:53:00.985272 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:00.985232 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/14242834-56aa-458b-bacc-bb5a4aa97937-node-exporter-textfile\") pod \"node-exporter-kf24r\" (UID: \"14242834-56aa-458b-bacc-bb5a4aa97937\") " pod="openshift-monitoring/node-exporter-kf24r" Nov 02 22:53:00.985272 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:00.985251 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/14242834-56aa-458b-bacc-bb5a4aa97937-node-exporter-wtmp\") pod \"node-exporter-kf24r\" (UID: \"14242834-56aa-458b-bacc-bb5a4aa97937\") " pod="openshift-monitoring/node-exporter-kf24r" Nov 02 22:53:00.985632 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:00.985289 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14242834-56aa-458b-bacc-bb5a4aa97937-sys\") pod \"node-exporter-kf24r\" (UID: \"14242834-56aa-458b-bacc-bb5a4aa97937\") " pod="openshift-monitoring/node-exporter-kf24r" Nov 02 22:53:00.985632 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:00.985370 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14242834-56aa-458b-bacc-bb5a4aa97937-sys\") pod \"node-exporter-kf24r\" (UID: \"14242834-56aa-458b-bacc-bb5a4aa97937\") " pod="openshift-monitoring/node-exporter-kf24r" Nov 02 22:53:00.985632 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:00.985407 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/14242834-56aa-458b-bacc-bb5a4aa97937-root\") pod \"node-exporter-kf24r\" (UID: \"14242834-56aa-458b-bacc-bb5a4aa97937\") " pod="openshift-monitoring/node-exporter-kf24r" Nov 02 22:53:00.985909 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:00.985749 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/14242834-56aa-458b-bacc-bb5a4aa97937-node-exporter-wtmp\") pod \"node-exporter-kf24r\" (UID: \"14242834-56aa-458b-bacc-bb5a4aa97937\") " pod="openshift-monitoring/node-exporter-kf24r" Nov 02 22:53:00.985973 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:00.985928 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/14242834-56aa-458b-bacc-bb5a4aa97937-node-exporter-textfile\") pod \"node-exporter-kf24r\" (UID: \"14242834-56aa-458b-bacc-bb5a4aa97937\") " pod="openshift-monitoring/node-exporter-kf24r" Nov 02 22:53:00.986029 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:00.985987 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14242834-56aa-458b-bacc-bb5a4aa97937-metrics-client-ca\") pod \"node-exporter-kf24r\" (UID: \"14242834-56aa-458b-bacc-bb5a4aa97937\") " pod="openshift-monitoring/node-exporter-kf24r" Nov 02 22:53:00.986252 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:00.986233 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/14242834-56aa-458b-bacc-bb5a4aa97937-node-exporter-accelerators-collector-config\") pod \"node-exporter-kf24r\" (UID: \"14242834-56aa-458b-bacc-bb5a4aa97937\") " pod="openshift-monitoring/node-exporter-kf24r" Nov 02 22:53:00.987941 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:00.987915 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14242834-56aa-458b-bacc-bb5a4aa97937-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kf24r\" (UID: \"14242834-56aa-458b-bacc-bb5a4aa97937\") " pod="openshift-monitoring/node-exporter-kf24r" Nov 02 22:53:00.988872 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:00.988849 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/14242834-56aa-458b-bacc-bb5a4aa97937-node-exporter-tls\") pod \"node-exporter-kf24r\" (UID: \"14242834-56aa-458b-bacc-bb5a4aa97937\") " pod="openshift-monitoring/node-exporter-kf24r" Nov 02 22:53:01.023726 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:01.023678 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rmgpt" Nov 02 22:53:01.049634 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:01.049554 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7rz6\" (UniqueName: \"kubernetes.io/projected/14242834-56aa-458b-bacc-bb5a4aa97937-kube-api-access-z7rz6\") pod \"node-exporter-kf24r\" (UID: \"14242834-56aa-458b-bacc-bb5a4aa97937\") " pod="openshift-monitoring/node-exporter-kf24r" Nov 02 22:53:01.093210 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:01.093110 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kf24r" Nov 02 22:53:01.105399 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:53:01.105361 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14242834_56aa_458b_bacc_bb5a4aa97937.slice/crio-16d17cf113d82b2219485d89a63eefa56f914d7afcb5368765ca649f22eee289 WatchSource:0}: Error finding container 16d17cf113d82b2219485d89a63eefa56f914d7afcb5368765ca649f22eee289: Status 404 returned error can't find the container with id 16d17cf113d82b2219485d89a63eefa56f914d7afcb5368765ca649f22eee289 Nov 02 22:53:01.383151 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:01.383078 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kf24r" event={"ID":"14242834-56aa-458b-bacc-bb5a4aa97937","Type":"ContainerStarted","Data":"16d17cf113d82b2219485d89a63eefa56f914d7afcb5368765ca649f22eee289"} Nov 02 22:53:01.959880 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:01.959842 5076 patch_prober.go:29] interesting pod/image-registry-7b5b8cd649-z4rbs container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 02 22:53:01.960346 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:01.959926 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" podUID="46eae5da-cf0b-4663-bdeb-f8220fa08c05" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:53:03.397038 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:03.397001 5076 generic.go:356] "Generic (PLEG): container finished" podID="14242834-56aa-458b-bacc-bb5a4aa97937" containerID="df2c12cfa39cb853dad023678f82aa8b1abe6cd14c84fe6e61489e934e25542a" exitCode=0 Nov 02 22:53:03.397505 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:03.397097 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kf24r" event={"ID":"14242834-56aa-458b-bacc-bb5a4aa97937","Type":"ContainerDied","Data":"df2c12cfa39cb853dad023678f82aa8b1abe6cd14c84fe6e61489e934e25542a"} Nov 02 22:53:04.404735 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:04.404692 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kf24r" event={"ID":"14242834-56aa-458b-bacc-bb5a4aa97937","Type":"ContainerStarted","Data":"9dde0ec419dc6c956aada0d9a7155918144000dec94c5b037a39f8ce62a941ae"} Nov 02 22:53:04.405199 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:04.404752 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kf24r" event={"ID":"14242834-56aa-458b-bacc-bb5a4aa97937","Type":"ContainerStarted","Data":"551b52e8d0c6cbdcd28f3920e7ca120750ebd317db794dc76f969f472202b59a"} Nov 02 22:53:04.518149 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:04.518093 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kf24r" podStartSLOduration=2.716518001 podStartE2EDuration="4.518079853s" totalImagesPullingTime="1.723450969s" totalInitContainerRuntime="78.110883ms" isStatefulPod=true podCreationTimestamp="2026-11-02 22:53:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:53:04.516476524 +0000 UTC m=+33.485008308" watchObservedRunningTime="2026-11-02 22:53:04.518079853 +0000 UTC m=+33.486611685" Nov 02 22:53:06.415306 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:06.415217 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-p44bf" event={"ID":"4e1fcec6-163f-4442-8444-19c2c00986f0","Type":"ContainerStarted","Data":"afeb37fd6b71657192b6ddbc53b0621c84f047fdbaf07cc6646345eb25c71fc4"} Nov 02 22:53:06.415804 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:06.415477 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-p44bf" Nov 02 22:53:06.433202 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:06.433165 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-p44bf" Nov 02 22:53:06.620252 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:06.620191 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-p44bf" podStartSLOduration=2.085680795 podStartE2EDuration="18.620176109s" totalImagesPullingTime="16.534495314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:52:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:53:06.572035551 +0000 UTC m=+35.540567356" watchObservedRunningTime="2026-11-02 22:53:06.620176109 +0000 UTC m=+35.588707890" Nov 02 22:53:08.705023 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:08.704992 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:53:09.885924 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:53:09.885869 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 22:53:11.331796 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:11.331706 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6fffcf7b58-2vvth" Nov 02 22:53:11.547802 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:11.547764 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b5b8cd649-z4rbs"] Nov 02 22:53:11.553692 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:11.553659 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:53:13.727683 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:13.727607 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-777f69544d-d75n6" podUID="0237eeb1-8c85-475e-8580-6dec324de705" containerName="registry" containerID="cri-o://22e3aa5b6126ab82576f8f963fd84c26599ce5c8be1bb36eb3a7b10915f2297f" gracePeriod=30 Nov 02 22:53:14.040991 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:14.040949 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:53:14.203732 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:14.202941 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0237eeb1-8c85-475e-8580-6dec324de705-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0237eeb1-8c85-475e-8580-6dec324de705-installation-pull-secrets\") pod \"0237eeb1-8c85-475e-8580-6dec324de705\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " Nov 02 22:53:14.203732 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:14.203023 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-registry-tls\" (UniqueName: \"kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-registry-tls\") pod \"0237eeb1-8c85-475e-8580-6dec324de705\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " Nov 02 22:53:14.203732 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:14.203089 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0237eeb1-8c85-475e-8580-6dec324de705-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0237eeb1-8c85-475e-8580-6dec324de705-trusted-ca\") pod \"0237eeb1-8c85-475e-8580-6dec324de705\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " Nov 02 22:53:14.203732 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:14.203119 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-kube-api-access-nwnxj\" (UniqueName: \"kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-kube-api-access-nwnxj\") pod \"0237eeb1-8c85-475e-8580-6dec324de705\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " Nov 02 22:53:14.203732 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:14.203152 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0237eeb1-8c85-475e-8580-6dec324de705-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0237eeb1-8c85-475e-8580-6dec324de705-image-registry-private-configuration\") pod \"0237eeb1-8c85-475e-8580-6dec324de705\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " Nov 02 22:53:14.203732 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:14.203203 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-bound-sa-token\") pod \"0237eeb1-8c85-475e-8580-6dec324de705\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " Nov 02 22:53:14.203732 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:14.203255 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0237eeb1-8c85-475e-8580-6dec324de705-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0237eeb1-8c85-475e-8580-6dec324de705-registry-certificates\") pod \"0237eeb1-8c85-475e-8580-6dec324de705\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " Nov 02 22:53:14.203732 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:14.203286 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0237eeb1-8c85-475e-8580-6dec324de705-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0237eeb1-8c85-475e-8580-6dec324de705-ca-trust-extracted\") pod \"0237eeb1-8c85-475e-8580-6dec324de705\" (UID: \"0237eeb1-8c85-475e-8580-6dec324de705\") " Nov 02 22:53:14.203732 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:14.203512 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0237eeb1-8c85-475e-8580-6dec324de705-trusted-ca" pod "0237eeb1-8c85-475e-8580-6dec324de705" (UID: "0237eeb1-8c85-475e-8580-6dec324de705"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:53:14.205835 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:14.204660 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0237eeb1-8c85-475e-8580-6dec324de705-registry-certificates" pod "0237eeb1-8c85-475e-8580-6dec324de705" (UID: "0237eeb1-8c85-475e-8580-6dec324de705"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:53:14.206865 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:14.206259 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-registry-tls" pod "0237eeb1-8c85-475e-8580-6dec324de705" (UID: "0237eeb1-8c85-475e-8580-6dec324de705"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:53:14.206865 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:14.206609 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0237eeb1-8c85-475e-8580-6dec324de705-image-registry-private-configuration" pod "0237eeb1-8c85-475e-8580-6dec324de705" (UID: "0237eeb1-8c85-475e-8580-6dec324de705"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:53:14.207831 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:14.207590 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0237eeb1-8c85-475e-8580-6dec324de705-installation-pull-secrets" pod "0237eeb1-8c85-475e-8580-6dec324de705" (UID: "0237eeb1-8c85-475e-8580-6dec324de705"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:53:14.207831 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:14.207697 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-kube-api-access-nwnxj" pod "0237eeb1-8c85-475e-8580-6dec324de705" (UID: "0237eeb1-8c85-475e-8580-6dec324de705"). InnerVolumeSpecName "kube-api-access-nwnxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:53:14.210734 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:14.210702 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-bound-sa-token" pod "0237eeb1-8c85-475e-8580-6dec324de705" (UID: "0237eeb1-8c85-475e-8580-6dec324de705"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:53:14.213586 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:14.213550 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0237eeb1-8c85-475e-8580-6dec324de705-ca-trust-extracted" pod "0237eeb1-8c85-475e-8580-6dec324de705" (UID: "0237eeb1-8c85-475e-8580-6dec324de705"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:53:14.304920 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:14.304827 5076 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0237eeb1-8c85-475e-8580-6dec324de705-installation-pull-secrets\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:14.304920 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:14.304864 5076 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-registry-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:14.304920 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:14.304880 5076 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0237eeb1-8c85-475e-8580-6dec324de705-trusted-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:14.305217 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:14.304949 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwnxj\" (UniqueName: \"kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-kube-api-access-nwnxj\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:14.305217 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:14.304967 5076 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0237eeb1-8c85-475e-8580-6dec324de705-image-registry-private-configuration\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:14.305217 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:14.304981 5076 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0237eeb1-8c85-475e-8580-6dec324de705-bound-sa-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:14.305217 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:14.304995 5076 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0237eeb1-8c85-475e-8580-6dec324de705-registry-certificates\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:14.305217 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:14.305008 5076 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0237eeb1-8c85-475e-8580-6dec324de705-ca-trust-extracted\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:14.453792 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:14.453754 5076 generic.go:356] "Generic (PLEG): container finished" podID="0237eeb1-8c85-475e-8580-6dec324de705" containerID="22e3aa5b6126ab82576f8f963fd84c26599ce5c8be1bb36eb3a7b10915f2297f" exitCode=0 Nov 02 22:53:14.453982 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:14.453837 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-777f69544d-d75n6" Nov 02 22:53:14.453982 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:14.453858 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-777f69544d-d75n6" event={"ID":"0237eeb1-8c85-475e-8580-6dec324de705","Type":"ContainerDied","Data":"22e3aa5b6126ab82576f8f963fd84c26599ce5c8be1bb36eb3a7b10915f2297f"} Nov 02 22:53:14.453982 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:14.453944 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-777f69544d-d75n6" event={"ID":"0237eeb1-8c85-475e-8580-6dec324de705","Type":"ContainerDied","Data":"f8cd1c3bc352c6b10bad63b1002882bb9e2547956cbd2a0ca1159436285a3a2e"} Nov 02 22:53:14.453982 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:14.453975 5076 scope.go:122] "RemoveContainer" containerID="22e3aa5b6126ab82576f8f963fd84c26599ce5c8be1bb36eb3a7b10915f2297f" Nov 02 22:53:14.465656 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:14.465594 5076 scope.go:122] "RemoveContainer" containerID="22e3aa5b6126ab82576f8f963fd84c26599ce5c8be1bb36eb3a7b10915f2297f" Nov 02 22:53:14.466103 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:53:14.466074 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22e3aa5b6126ab82576f8f963fd84c26599ce5c8be1bb36eb3a7b10915f2297f\": container with ID starting with 22e3aa5b6126ab82576f8f963fd84c26599ce5c8be1bb36eb3a7b10915f2297f not found: ID does not exist" containerID="22e3aa5b6126ab82576f8f963fd84c26599ce5c8be1bb36eb3a7b10915f2297f" Nov 02 22:53:14.466212 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:14.466115 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22e3aa5b6126ab82576f8f963fd84c26599ce5c8be1bb36eb3a7b10915f2297f"} err="failed to get container status \"22e3aa5b6126ab82576f8f963fd84c26599ce5c8be1bb36eb3a7b10915f2297f\": rpc error: code = NotFound desc = could not find container \"22e3aa5b6126ab82576f8f963fd84c26599ce5c8be1bb36eb3a7b10915f2297f\": container with ID starting with 22e3aa5b6126ab82576f8f963fd84c26599ce5c8be1bb36eb3a7b10915f2297f not found: ID does not exist" Nov 02 22:53:14.609878 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:14.609783 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-777f69544d-d75n6"] Nov 02 22:53:14.656505 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:14.656476 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-777f69544d-d75n6"] Nov 02 22:53:15.149788 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:15.147016 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0237eeb1-8c85-475e-8580-6dec324de705" path="/var/lib/kubelet/pods/0237eeb1-8c85-475e-8580-6dec324de705/volumes" Nov 02 22:53:36.572124 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:36.572074 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" podUID="46eae5da-cf0b-4663-bdeb-f8220fa08c05" containerName="registry" containerID="cri-o://c462a03a6914a2369de0cff964427ecd1ba7ef8f0c1085402d6e68e66734ba09" gracePeriod=30 Nov 02 22:53:36.863477 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:36.863452 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:53:36.870399 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:36.870374 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-registry-tls\" (UniqueName: \"kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-registry-tls\") pod \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " Nov 02 22:53:36.870530 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:36.870413 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46eae5da-cf0b-4663-bdeb-f8220fa08c05-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/46eae5da-cf0b-4663-bdeb-f8220fa08c05-installation-pull-secrets\") pod \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " Nov 02 22:53:36.870530 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:36.870463 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46eae5da-cf0b-4663-bdeb-f8220fa08c05-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/46eae5da-cf0b-4663-bdeb-f8220fa08c05-image-registry-private-configuration\") pod \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " Nov 02 22:53:36.870530 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:36.870510 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46eae5da-cf0b-4663-bdeb-f8220fa08c05-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/46eae5da-cf0b-4663-bdeb-f8220fa08c05-ca-trust-extracted\") pod \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " Nov 02 22:53:36.870687 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:36.870549 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46eae5da-cf0b-4663-bdeb-f8220fa08c05-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/46eae5da-cf0b-4663-bdeb-f8220fa08c05-registry-certificates\") pod \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " Nov 02 22:53:36.870687 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:36.870606 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46eae5da-cf0b-4663-bdeb-f8220fa08c05-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/46eae5da-cf0b-4663-bdeb-f8220fa08c05-trusted-ca\") pod \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " Nov 02 22:53:36.870687 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:36.870637 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-kube-api-access-nkzvb\" (UniqueName: \"kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-kube-api-access-nkzvb\") pod \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " Nov 02 22:53:36.870833 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:36.870688 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-bound-sa-token\") pod \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\" (UID: \"46eae5da-cf0b-4663-bdeb-f8220fa08c05\") " Nov 02 22:53:36.871059 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:36.870996 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46eae5da-cf0b-4663-bdeb-f8220fa08c05-registry-certificates" pod "46eae5da-cf0b-4663-bdeb-f8220fa08c05" (UID: "46eae5da-cf0b-4663-bdeb-f8220fa08c05"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:53:36.871463 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:36.871367 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46eae5da-cf0b-4663-bdeb-f8220fa08c05-trusted-ca" pod "46eae5da-cf0b-4663-bdeb-f8220fa08c05" (UID: "46eae5da-cf0b-4663-bdeb-f8220fa08c05"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:53:36.872921 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:36.872855 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46eae5da-cf0b-4663-bdeb-f8220fa08c05-image-registry-private-configuration" pod "46eae5da-cf0b-4663-bdeb-f8220fa08c05" (UID: "46eae5da-cf0b-4663-bdeb-f8220fa08c05"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:53:36.873334 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:36.872990 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46eae5da-cf0b-4663-bdeb-f8220fa08c05-installation-pull-secrets" pod "46eae5da-cf0b-4663-bdeb-f8220fa08c05" (UID: "46eae5da-cf0b-4663-bdeb-f8220fa08c05"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:53:36.873334 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:36.873056 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-registry-tls" pod "46eae5da-cf0b-4663-bdeb-f8220fa08c05" (UID: "46eae5da-cf0b-4663-bdeb-f8220fa08c05"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:53:36.873334 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:36.873214 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-bound-sa-token" pod "46eae5da-cf0b-4663-bdeb-f8220fa08c05" (UID: "46eae5da-cf0b-4663-bdeb-f8220fa08c05"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:53:36.873334 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:36.873339 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-kube-api-access-nkzvb" pod "46eae5da-cf0b-4663-bdeb-f8220fa08c05" (UID: "46eae5da-cf0b-4663-bdeb-f8220fa08c05"). InnerVolumeSpecName "kube-api-access-nkzvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:53:36.879788 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:36.879760 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46eae5da-cf0b-4663-bdeb-f8220fa08c05-ca-trust-extracted" pod "46eae5da-cf0b-4663-bdeb-f8220fa08c05" (UID: "46eae5da-cf0b-4663-bdeb-f8220fa08c05"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:53:36.971416 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:36.971381 5076 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/46eae5da-cf0b-4663-bdeb-f8220fa08c05-trusted-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:36.971416 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:36.971413 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkzvb\" (UniqueName: \"kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-kube-api-access-nkzvb\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:36.971624 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:36.971428 5076 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-bound-sa-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:36.971624 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:36.971440 5076 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/46eae5da-cf0b-4663-bdeb-f8220fa08c05-registry-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:36.971624 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:36.971454 5076 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/46eae5da-cf0b-4663-bdeb-f8220fa08c05-installation-pull-secrets\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:36.971624 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:36.971466 5076 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/46eae5da-cf0b-4663-bdeb-f8220fa08c05-image-registry-private-configuration\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:36.971624 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:36.971479 5076 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/46eae5da-cf0b-4663-bdeb-f8220fa08c05-ca-trust-extracted\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:36.971624 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:36.971491 5076 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/46eae5da-cf0b-4663-bdeb-f8220fa08c05-registry-certificates\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:37.550406 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:37.550365 5076 generic.go:356] "Generic (PLEG): container finished" podID="46eae5da-cf0b-4663-bdeb-f8220fa08c05" containerID="c462a03a6914a2369de0cff964427ecd1ba7ef8f0c1085402d6e68e66734ba09" exitCode=0 Nov 02 22:53:37.550570 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:37.550428 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" Nov 02 22:53:37.550570 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:37.550427 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" event={"ID":"46eae5da-cf0b-4663-bdeb-f8220fa08c05","Type":"ContainerDied","Data":"c462a03a6914a2369de0cff964427ecd1ba7ef8f0c1085402d6e68e66734ba09"} Nov 02 22:53:37.550570 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:37.550544 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b5b8cd649-z4rbs" event={"ID":"46eae5da-cf0b-4663-bdeb-f8220fa08c05","Type":"ContainerDied","Data":"b09b34b59f5ba9a0c00d160adb7cf7a2de548573d0d18bf39fc7b0d1e24e67e5"} Nov 02 22:53:37.550570 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:37.550568 5076 scope.go:122] "RemoveContainer" containerID="c462a03a6914a2369de0cff964427ecd1ba7ef8f0c1085402d6e68e66734ba09" Nov 02 22:53:37.558739 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:37.558719 5076 scope.go:122] "RemoveContainer" containerID="c462a03a6914a2369de0cff964427ecd1ba7ef8f0c1085402d6e68e66734ba09" Nov 02 22:53:37.559025 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:53:37.559006 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c462a03a6914a2369de0cff964427ecd1ba7ef8f0c1085402d6e68e66734ba09\": container with ID starting with c462a03a6914a2369de0cff964427ecd1ba7ef8f0c1085402d6e68e66734ba09 not found: ID does not exist" containerID="c462a03a6914a2369de0cff964427ecd1ba7ef8f0c1085402d6e68e66734ba09" Nov 02 22:53:37.559076 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:37.559036 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c462a03a6914a2369de0cff964427ecd1ba7ef8f0c1085402d6e68e66734ba09"} err="failed to get container status \"c462a03a6914a2369de0cff964427ecd1ba7ef8f0c1085402d6e68e66734ba09\": rpc error: code = NotFound desc = could not find container \"c462a03a6914a2369de0cff964427ecd1ba7ef8f0c1085402d6e68e66734ba09\": container with ID starting with c462a03a6914a2369de0cff964427ecd1ba7ef8f0c1085402d6e68e66734ba09 not found: ID does not exist" Nov 02 22:53:37.753382 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:37.753351 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b5b8cd649-z4rbs"] Nov 02 22:53:37.800916 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:37.800837 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7b5b8cd649-z4rbs"] Nov 02 22:53:39.145640 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:39.145606 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46eae5da-cf0b-4663-bdeb-f8220fa08c05" path="/var/lib/kubelet/pods/46eae5da-cf0b-4663-bdeb-f8220fa08c05/volumes" Nov 02 22:53:39.882080 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:53:39.882045 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 22:53:52.613973 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:52.613938 5076 generic.go:356] "Generic (PLEG): container finished" podID="b858466f-3dcd-419c-9495-4dbb89e8d51b" containerID="8495045bd8628a15e3796c689e1715ce7627d22e65221455858e5862a91aed6c" exitCode=0 Nov 02 22:53:52.614522 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:52.614016 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-xszmw" event={"ID":"b858466f-3dcd-419c-9495-4dbb89e8d51b","Type":"ContainerDied","Data":"8495045bd8628a15e3796c689e1715ce7627d22e65221455858e5862a91aed6c"} Nov 02 22:53:52.614522 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:52.614444 5076 scope.go:122] "RemoveContainer" containerID="8495045bd8628a15e3796c689e1715ce7627d22e65221455858e5862a91aed6c" Nov 02 22:53:53.619823 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:53.619786 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-xszmw" event={"ID":"b858466f-3dcd-419c-9495-4dbb89e8d51b","Type":"ContainerStarted","Data":"be24f06ff5010711c652430de5594e782311a1d0429618e52d4739a9ec9a6a51"} Nov 02 22:53:53.842991 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:53.842954 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-lx8nq-mmb4p"] Nov 02 22:53:53.843174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:53.843154 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-lx8nq-mmb4p" podUID="d12c211b-f607-43b9-b621-12f061f109d9" containerName="insights-gathering" containerID="cri-o://29d97269a85c034b372c9aa93b7ff5f2295abd6535fe741e8ab8c382e87f6393" gracePeriod=30 Nov 02 22:53:54.098213 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:54.098188 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lx8nq-mmb4p" Nov 02 22:53:54.208811 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:54.208735 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d12c211b-f607-43b9-b621-12f061f109d9-kube-api-access-bx9mw\" (UniqueName: \"kubernetes.io/projected/d12c211b-f607-43b9-b621-12f061f109d9-kube-api-access-bx9mw\") pod \"d12c211b-f607-43b9-b621-12f061f109d9\" (UID: \"d12c211b-f607-43b9-b621-12f061f109d9\") " Nov 02 22:53:54.208991 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:54.208820 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d12c211b-f607-43b9-b621-12f061f109d9-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d12c211b-f607-43b9-b621-12f061f109d9-service-ca-bundle\") pod \"d12c211b-f607-43b9-b621-12f061f109d9\" (UID: \"d12c211b-f607-43b9-b621-12f061f109d9\") " Nov 02 22:53:54.208991 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:54.208869 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d12c211b-f607-43b9-b621-12f061f109d9-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d12c211b-f607-43b9-b621-12f061f109d9-trusted-ca-bundle\") pod \"d12c211b-f607-43b9-b621-12f061f109d9\" (UID: \"d12c211b-f607-43b9-b621-12f061f109d9\") " Nov 02 22:53:54.208991 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:54.208950 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d12c211b-f607-43b9-b621-12f061f109d9-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d12c211b-f607-43b9-b621-12f061f109d9-archives-path\") pod \"d12c211b-f607-43b9-b621-12f061f109d9\" (UID: \"d12c211b-f607-43b9-b621-12f061f109d9\") " Nov 02 22:53:54.209353 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:54.209277 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d12c211b-f607-43b9-b621-12f061f109d9-service-ca-bundle" pod "d12c211b-f607-43b9-b621-12f061f109d9" (UID: "d12c211b-f607-43b9-b621-12f061f109d9"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:53:54.209494 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:54.209423 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12c211b-f607-43b9-b621-12f061f109d9-archives-path" pod "d12c211b-f607-43b9-b621-12f061f109d9" (UID: "d12c211b-f607-43b9-b621-12f061f109d9"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:53:54.209821 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:54.209785 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d12c211b-f607-43b9-b621-12f061f109d9-trusted-ca-bundle" pod "d12c211b-f607-43b9-b621-12f061f109d9" (UID: "d12c211b-f607-43b9-b621-12f061f109d9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:53:54.211469 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:54.211445 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d12c211b-f607-43b9-b621-12f061f109d9-kube-api-access-bx9mw" pod "d12c211b-f607-43b9-b621-12f061f109d9" (UID: "d12c211b-f607-43b9-b621-12f061f109d9"). InnerVolumeSpecName "kube-api-access-bx9mw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:53:54.309491 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:54.309454 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx9mw\" (UniqueName: \"kubernetes.io/projected/d12c211b-f607-43b9-b621-12f061f109d9-kube-api-access-bx9mw\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:54.309491 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:54.309484 5076 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d12c211b-f607-43b9-b621-12f061f109d9-service-ca-bundle\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:54.309491 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:54.309495 5076 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d12c211b-f607-43b9-b621-12f061f109d9-trusted-ca-bundle\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:54.309711 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:54.309505 5076 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d12c211b-f607-43b9-b621-12f061f109d9-archives-path\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:54.624810 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:54.624735 5076 generic.go:356] "Generic (PLEG): container finished" podID="d12c211b-f607-43b9-b621-12f061f109d9" containerID="29d97269a85c034b372c9aa93b7ff5f2295abd6535fe741e8ab8c382e87f6393" exitCode=2 Nov 02 22:53:54.624810 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:54.624780 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lx8nq-mmb4p" Nov 02 22:53:54.624810 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:54.624806 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lx8nq-mmb4p" event={"ID":"d12c211b-f607-43b9-b621-12f061f109d9","Type":"ContainerDied","Data":"29d97269a85c034b372c9aa93b7ff5f2295abd6535fe741e8ab8c382e87f6393"} Nov 02 22:53:54.625309 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:54.624845 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lx8nq-mmb4p" event={"ID":"d12c211b-f607-43b9-b621-12f061f109d9","Type":"ContainerDied","Data":"923bdca363bdecec9681204d8c7d61a686ad35aad15b349b72375be1ca7ac71e"} Nov 02 22:53:54.625309 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:54.624862 5076 scope.go:122] "RemoveContainer" containerID="29d97269a85c034b372c9aa93b7ff5f2295abd6535fe741e8ab8c382e87f6393" Nov 02 22:53:54.632827 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:54.632812 5076 scope.go:122] "RemoveContainer" containerID="29d97269a85c034b372c9aa93b7ff5f2295abd6535fe741e8ab8c382e87f6393" Nov 02 22:53:54.633111 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:53:54.633095 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29d97269a85c034b372c9aa93b7ff5f2295abd6535fe741e8ab8c382e87f6393\": container with ID starting with 29d97269a85c034b372c9aa93b7ff5f2295abd6535fe741e8ab8c382e87f6393 not found: ID does not exist" containerID="29d97269a85c034b372c9aa93b7ff5f2295abd6535fe741e8ab8c382e87f6393" Nov 02 22:53:54.633162 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:54.633123 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29d97269a85c034b372c9aa93b7ff5f2295abd6535fe741e8ab8c382e87f6393"} err="failed to get container status \"29d97269a85c034b372c9aa93b7ff5f2295abd6535fe741e8ab8c382e87f6393\": rpc error: code = NotFound desc = could not find container \"29d97269a85c034b372c9aa93b7ff5f2295abd6535fe741e8ab8c382e87f6393\": container with ID starting with 29d97269a85c034b372c9aa93b7ff5f2295abd6535fe741e8ab8c382e87f6393 not found: ID does not exist" Nov 02 22:53:54.779711 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:54.779673 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-lx8nq-mmb4p"] Nov 02 22:53:54.827330 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:54.827296 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-lx8nq-mmb4p"] Nov 02 22:53:55.145664 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:55.145630 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d12c211b-f607-43b9-b621-12f061f109d9" path="/var/lib/kubelet/pods/d12c211b-f607-43b9-b621-12f061f109d9/volumes" Nov 02 22:53:57.639673 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:57.639636 5076 generic.go:356] "Generic (PLEG): container finished" podID="60832943-bf1f-4685-be00-08152c423b08" containerID="207bec755b31eb4ef158daaf761bd0e8fe353dd48459ead26c5aa9b9ab0fe67a" exitCode=0 Nov 02 22:53:57.640117 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:57.639709 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p66wt" event={"ID":"60832943-bf1f-4685-be00-08152c423b08","Type":"ContainerDied","Data":"207bec755b31eb4ef158daaf761bd0e8fe353dd48459ead26c5aa9b9ab0fe67a"} Nov 02 22:53:57.640117 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:57.640048 5076 scope.go:122] "RemoveContainer" containerID="207bec755b31eb4ef158daaf761bd0e8fe353dd48459ead26c5aa9b9ab0fe67a" Nov 02 22:53:58.647022 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:53:58.646989 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p66wt" event={"ID":"60832943-bf1f-4685-be00-08152c423b08","Type":"ContainerStarted","Data":"275045a286471ac8f9716b64f35d41e1b52869e7c4e276267ab8804c82c669f7"} Nov 02 22:54:09.881611 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:54:09.881566 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 22:54:39.881628 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:54:39.881585 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 22:55:09.881984 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:55:09.881880 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 22:55:39.882657 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:55:39.882614 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 22:56:09.867880 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:09.867841 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg"] Nov 02 22:56:09.868437 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:09.868414 5076 cpu_manager.go:414] "containerMap: removing container" podUID="0237eeb1-8c85-475e-8580-6dec324de705" containerName="registry" Nov 02 22:56:09.868478 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:09.868443 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0237eeb1-8c85-475e-8580-6dec324de705" containerName="registry" Nov 02 22:56:09.868478 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:09.868458 5076 cpu_manager.go:414] "containerMap: removing container" podUID="46eae5da-cf0b-4663-bdeb-f8220fa08c05" containerName="registry" Nov 02 22:56:09.868478 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:09.868469 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46eae5da-cf0b-4663-bdeb-f8220fa08c05" containerName="registry" Nov 02 22:56:09.868567 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:09.868510 5076 cpu_manager.go:414] "containerMap: removing container" podUID="d12c211b-f607-43b9-b621-12f061f109d9" containerName="insights-gathering" Nov 02 22:56:09.868567 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:09.868519 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d12c211b-f607-43b9-b621-12f061f109d9" containerName="insights-gathering" Nov 02 22:56:09.868628 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:09.868623 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="0237eeb1-8c85-475e-8580-6dec324de705" containerName="registry" Nov 02 22:56:09.868658 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:09.868633 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="46eae5da-cf0b-4663-bdeb-f8220fa08c05" containerName="registry" Nov 02 22:56:09.868658 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:09.868645 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="d12c211b-f607-43b9-b621-12f061f109d9" containerName="insights-gathering" Nov 02 22:56:09.870807 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:09.870787 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg" Nov 02 22:56:09.884875 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:56:09.884826 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 22:56:09.905602 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:09.905571 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76adda1-d0ff-456d-a420-24babb08880b-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg\" (UID: \"c76adda1-d0ff-456d-a420-24babb08880b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg" Nov 02 22:56:09.905751 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:09.905610 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76adda1-d0ff-456d-a420-24babb08880b-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg\" (UID: \"c76adda1-d0ff-456d-a420-24babb08880b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg" Nov 02 22:56:09.905751 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:09.905634 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2g49\" (UniqueName: \"kubernetes.io/projected/c76adda1-d0ff-456d-a420-24babb08880b-kube-api-access-f2g49\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg\" (UID: \"c76adda1-d0ff-456d-a420-24babb08880b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg" Nov 02 22:56:09.905868 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:09.905783 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76adda1-d0ff-456d-a420-24babb08880b-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg\" (UID: \"c76adda1-d0ff-456d-a420-24babb08880b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg" Nov 02 22:56:09.905868 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:09.905827 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76adda1-d0ff-456d-a420-24babb08880b-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg\" (UID: \"c76adda1-d0ff-456d-a420-24babb08880b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg" Nov 02 22:56:09.905992 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:09.905866 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76adda1-d0ff-456d-a420-24babb08880b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg\" (UID: \"c76adda1-d0ff-456d-a420-24babb08880b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg" Nov 02 22:56:09.906042 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:09.905987 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76adda1-d0ff-456d-a420-24babb08880b-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg\" (UID: \"c76adda1-d0ff-456d-a420-24babb08880b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg" Nov 02 22:56:09.906042 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:09.906021 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76adda1-d0ff-456d-a420-24babb08880b-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg\" (UID: \"c76adda1-d0ff-456d-a420-24babb08880b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg" Nov 02 22:56:09.906130 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:09.906052 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76adda1-d0ff-456d-a420-24babb08880b-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg\" (UID: \"c76adda1-d0ff-456d-a420-24babb08880b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg" Nov 02 22:56:09.920220 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:09.920189 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 22:56:09.948951 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:09.948915 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-pwfp8\"" Nov 02 22:56:09.968772 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:09.968739 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg"] Nov 02 22:56:10.006545 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:10.006510 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76adda1-d0ff-456d-a420-24babb08880b-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg\" (UID: \"c76adda1-d0ff-456d-a420-24babb08880b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg" Nov 02 22:56:10.006545 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:10.006547 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76adda1-d0ff-456d-a420-24babb08880b-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg\" (UID: \"c76adda1-d0ff-456d-a420-24babb08880b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg" Nov 02 22:56:10.006795 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:10.006571 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2g49\" (UniqueName: \"kubernetes.io/projected/c76adda1-d0ff-456d-a420-24babb08880b-kube-api-access-f2g49\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg\" (UID: \"c76adda1-d0ff-456d-a420-24babb08880b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg" Nov 02 22:56:10.006795 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:10.006780 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76adda1-d0ff-456d-a420-24babb08880b-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg\" (UID: \"c76adda1-d0ff-456d-a420-24babb08880b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg" Nov 02 22:56:10.006951 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:10.006818 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76adda1-d0ff-456d-a420-24babb08880b-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg\" (UID: \"c76adda1-d0ff-456d-a420-24babb08880b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg" Nov 02 22:56:10.007016 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:10.006973 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76adda1-d0ff-456d-a420-24babb08880b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg\" (UID: \"c76adda1-d0ff-456d-a420-24babb08880b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg" Nov 02 22:56:10.007072 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:10.007027 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76adda1-d0ff-456d-a420-24babb08880b-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg\" (UID: \"c76adda1-d0ff-456d-a420-24babb08880b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg" Nov 02 22:56:10.007072 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:10.007036 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76adda1-d0ff-456d-a420-24babb08880b-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg\" (UID: \"c76adda1-d0ff-456d-a420-24babb08880b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg" Nov 02 22:56:10.007072 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:10.007057 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76adda1-d0ff-456d-a420-24babb08880b-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg\" (UID: \"c76adda1-d0ff-456d-a420-24babb08880b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg" Nov 02 22:56:10.007220 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:10.007105 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76adda1-d0ff-456d-a420-24babb08880b-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg\" (UID: \"c76adda1-d0ff-456d-a420-24babb08880b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg" Nov 02 22:56:10.007220 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:10.007151 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76adda1-d0ff-456d-a420-24babb08880b-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg\" (UID: \"c76adda1-d0ff-456d-a420-24babb08880b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg" Nov 02 22:56:10.007328 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:10.007232 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76adda1-d0ff-456d-a420-24babb08880b-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg\" (UID: \"c76adda1-d0ff-456d-a420-24babb08880b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg" Nov 02 22:56:10.007424 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:10.007404 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76adda1-d0ff-456d-a420-24babb08880b-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg\" (UID: \"c76adda1-d0ff-456d-a420-24babb08880b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg" Nov 02 22:56:10.007702 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:10.007671 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76adda1-d0ff-456d-a420-24babb08880b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg\" (UID: \"c76adda1-d0ff-456d-a420-24babb08880b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg" Nov 02 22:56:10.009118 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:10.009092 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76adda1-d0ff-456d-a420-24babb08880b-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg\" (UID: \"c76adda1-d0ff-456d-a420-24babb08880b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg" Nov 02 22:56:10.009219 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:10.009195 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76adda1-d0ff-456d-a420-24babb08880b-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg\" (UID: \"c76adda1-d0ff-456d-a420-24babb08880b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg" Nov 02 22:56:10.060617 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:10.060584 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76adda1-d0ff-456d-a420-24babb08880b-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg\" (UID: \"c76adda1-d0ff-456d-a420-24babb08880b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg" Nov 02 22:56:10.060883 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:10.060859 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2g49\" (UniqueName: \"kubernetes.io/projected/c76adda1-d0ff-456d-a420-24babb08880b-kube-api-access-f2g49\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg\" (UID: \"c76adda1-d0ff-456d-a420-24babb08880b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg" Nov 02 22:56:10.182924 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:10.182820 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg" Nov 02 22:56:10.590191 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:10.590156 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg"] Nov 02 22:56:10.627605 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:56:10.627578 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc76adda1_d0ff_456d_a420_24babb08880b.slice/crio-d2401c485c6261797478d6ca3fdda6e1f65d1bbfeba5cc07b040d0b9e69d3d3b WatchSource:0}: Error finding container d2401c485c6261797478d6ca3fdda6e1f65d1bbfeba5cc07b040d0b9e69d3d3b: Status 404 returned error can't find the container with id d2401c485c6261797478d6ca3fdda6e1f65d1bbfeba5cc07b040d0b9e69d3d3b Nov 02 22:56:11.158398 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:11.158327 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg" event={"ID":"c76adda1-d0ff-456d-a420-24babb08880b","Type":"ContainerStarted","Data":"d2401c485c6261797478d6ca3fdda6e1f65d1bbfeba5cc07b040d0b9e69d3d3b"} Nov 02 22:56:15.691613 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:15.691585 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:56:15.691874 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:15.691661 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:56:15.691874 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:15.691696 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:56:16.179910 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:16.179808 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg" event={"ID":"c76adda1-d0ff-456d-a420-24babb08880b","Type":"ContainerStarted","Data":"e434fa915d54662a477951769187ca72c689002b33c703f1976c3dea1d824f28"} Nov 02 22:56:16.183230 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:16.183208 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg" Nov 02 22:56:16.187768 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:16.187744 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg" Nov 02 22:56:16.386358 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:16.386302 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg" podStartSLOduration=2.324960299 podStartE2EDuration="7.386287839s" totalImagesPullingTime="5.06132754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:56:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:56:16.338106514 +0000 UTC m=+225.306638301" watchObservedRunningTime="2026-11-02 22:56:16.386287839 +0000 UTC m=+225.354819622" Nov 02 22:56:17.183957 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:17.183923 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg" Nov 02 22:56:17.185000 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:17.184980 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg" Nov 02 22:56:39.882605 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:56:39.882568 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 22:56:49.677367 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:49.677328 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-s7pdf"] Nov 02 22:56:49.680851 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:49.680836 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-s7pdf" Nov 02 22:56:49.736569 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:49.736529 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 22:56:49.736718 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:49.736604 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 22:56:49.736807 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:49.736787 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 22:56:49.736862 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:49.736789 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-lpc6c\"" Nov 02 22:56:49.736862 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:49.736798 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 22:56:49.783905 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:49.783867 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 22:56:49.785700 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:49.785675 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-s7pdf"] Nov 02 22:56:49.838217 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:49.838182 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e39403cf-c60c-4ec5-b557-70af3a70e2a6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7858c78b8d-s7pdf\" (UID: \"e39403cf-c60c-4ec5-b557-70af3a70e2a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-s7pdf" Nov 02 22:56:49.838381 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:49.838249 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e39403cf-c60c-4ec5-b557-70af3a70e2a6-tmp\") pod \"opendatahub-operator-controller-manager-7858c78b8d-s7pdf\" (UID: \"e39403cf-c60c-4ec5-b557-70af3a70e2a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-s7pdf" Nov 02 22:56:49.838381 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:49.838357 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld6bb\" (UniqueName: \"kubernetes.io/projected/e39403cf-c60c-4ec5-b557-70af3a70e2a6-kube-api-access-ld6bb\") pod \"opendatahub-operator-controller-manager-7858c78b8d-s7pdf\" (UID: \"e39403cf-c60c-4ec5-b557-70af3a70e2a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-s7pdf" Nov 02 22:56:49.838484 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:49.838391 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e39403cf-c60c-4ec5-b557-70af3a70e2a6-webhook-cert\") pod \"opendatahub-operator-controller-manager-7858c78b8d-s7pdf\" (UID: \"e39403cf-c60c-4ec5-b557-70af3a70e2a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-s7pdf" Nov 02 22:56:49.838519 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:49.838475 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e39403cf-c60c-4ec5-b557-70af3a70e2a6-metrics-cert\") pod \"opendatahub-operator-controller-manager-7858c78b8d-s7pdf\" (UID: \"e39403cf-c60c-4ec5-b557-70af3a70e2a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-s7pdf" Nov 02 22:56:49.838556 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:49.838517 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e39403cf-c60c-4ec5-b557-70af3a70e2a6-manifests\") pod \"opendatahub-operator-controller-manager-7858c78b8d-s7pdf\" (UID: \"e39403cf-c60c-4ec5-b557-70af3a70e2a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-s7pdf" Nov 02 22:56:49.939378 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:49.939350 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld6bb\" (UniqueName: \"kubernetes.io/projected/e39403cf-c60c-4ec5-b557-70af3a70e2a6-kube-api-access-ld6bb\") pod \"opendatahub-operator-controller-manager-7858c78b8d-s7pdf\" (UID: \"e39403cf-c60c-4ec5-b557-70af3a70e2a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-s7pdf" Nov 02 22:56:49.939568 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:49.939392 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e39403cf-c60c-4ec5-b557-70af3a70e2a6-webhook-cert\") pod \"opendatahub-operator-controller-manager-7858c78b8d-s7pdf\" (UID: \"e39403cf-c60c-4ec5-b557-70af3a70e2a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-s7pdf" Nov 02 22:56:49.939568 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:49.939430 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e39403cf-c60c-4ec5-b557-70af3a70e2a6-metrics-cert\") pod \"opendatahub-operator-controller-manager-7858c78b8d-s7pdf\" (UID: \"e39403cf-c60c-4ec5-b557-70af3a70e2a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-s7pdf" Nov 02 22:56:49.939568 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:49.939464 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e39403cf-c60c-4ec5-b557-70af3a70e2a6-manifests\") pod \"opendatahub-operator-controller-manager-7858c78b8d-s7pdf\" (UID: \"e39403cf-c60c-4ec5-b557-70af3a70e2a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-s7pdf" Nov 02 22:56:49.939568 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:49.939507 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e39403cf-c60c-4ec5-b557-70af3a70e2a6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7858c78b8d-s7pdf\" (UID: \"e39403cf-c60c-4ec5-b557-70af3a70e2a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-s7pdf" Nov 02 22:56:49.939568 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:49.939542 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e39403cf-c60c-4ec5-b557-70af3a70e2a6-tmp\") pod \"opendatahub-operator-controller-manager-7858c78b8d-s7pdf\" (UID: \"e39403cf-c60c-4ec5-b557-70af3a70e2a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-s7pdf" Nov 02 22:56:49.939968 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:49.939938 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e39403cf-c60c-4ec5-b557-70af3a70e2a6-manifests\") pod \"opendatahub-operator-controller-manager-7858c78b8d-s7pdf\" (UID: \"e39403cf-c60c-4ec5-b557-70af3a70e2a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-s7pdf" Nov 02 22:56:49.941942 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:49.941911 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e39403cf-c60c-4ec5-b557-70af3a70e2a6-tmp\") pod \"opendatahub-operator-controller-manager-7858c78b8d-s7pdf\" (UID: \"e39403cf-c60c-4ec5-b557-70af3a70e2a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-s7pdf" Nov 02 22:56:49.942053 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:49.941972 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e39403cf-c60c-4ec5-b557-70af3a70e2a6-webhook-cert\") pod \"opendatahub-operator-controller-manager-7858c78b8d-s7pdf\" (UID: \"e39403cf-c60c-4ec5-b557-70af3a70e2a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-s7pdf" Nov 02 22:56:49.942053 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:49.942034 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e39403cf-c60c-4ec5-b557-70af3a70e2a6-metrics-cert\") pod \"opendatahub-operator-controller-manager-7858c78b8d-s7pdf\" (UID: \"e39403cf-c60c-4ec5-b557-70af3a70e2a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-s7pdf" Nov 02 22:56:49.942168 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:49.942125 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e39403cf-c60c-4ec5-b557-70af3a70e2a6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7858c78b8d-s7pdf\" (UID: \"e39403cf-c60c-4ec5-b557-70af3a70e2a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-s7pdf" Nov 02 22:56:49.995324 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:49.995292 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld6bb\" (UniqueName: \"kubernetes.io/projected/e39403cf-c60c-4ec5-b557-70af3a70e2a6-kube-api-access-ld6bb\") pod \"opendatahub-operator-controller-manager-7858c78b8d-s7pdf\" (UID: \"e39403cf-c60c-4ec5-b557-70af3a70e2a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-s7pdf" Nov 02 22:56:50.295650 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:50.295564 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-s7pdf" Nov 02 22:56:50.818550 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:50.818522 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-s7pdf"] Nov 02 22:56:50.820429 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:56:50.820399 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode39403cf_c60c_4ec5_b557_70af3a70e2a6.slice/crio-d97394ac7eb7aeece0baa9949d9d2fd5374c246fce85040d1822e37f357b7dcc WatchSource:0}: Error finding container d97394ac7eb7aeece0baa9949d9d2fd5374c246fce85040d1822e37f357b7dcc: Status 404 returned error can't find the container with id d97394ac7eb7aeece0baa9949d9d2fd5374c246fce85040d1822e37f357b7dcc Nov 02 22:56:51.324208 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:51.324167 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-s7pdf" event={"ID":"e39403cf-c60c-4ec5-b557-70af3a70e2a6","Type":"ContainerStarted","Data":"d97394ac7eb7aeece0baa9949d9d2fd5374c246fce85040d1822e37f357b7dcc"} Nov 02 22:56:55.343717 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:55.343631 5076 generic.go:356] "Generic (PLEG): container finished" podID="e39403cf-c60c-4ec5-b557-70af3a70e2a6" containerID="aed7b62a4f49f0934297e07eec1010be59db83d0807c87e19ab99339927b2f0e" exitCode=0 Nov 02 22:56:55.344110 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:55.343721 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-s7pdf" event={"ID":"e39403cf-c60c-4ec5-b557-70af3a70e2a6","Type":"ContainerDied","Data":"aed7b62a4f49f0934297e07eec1010be59db83d0807c87e19ab99339927b2f0e"} Nov 02 22:56:56.349837 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:56.349798 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-s7pdf" event={"ID":"e39403cf-c60c-4ec5-b557-70af3a70e2a6","Type":"ContainerStarted","Data":"594f25141d5aca80e2bb37d19af975672f4cc56852a020d189e0d74bff071414"} Nov 02 22:56:56.350328 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:56.349941 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-s7pdf" Nov 02 22:56:56.478989 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:56:56.478874 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-s7pdf" podStartSLOduration=2.485515691 podStartE2EDuration="7.478858871s" totalImagesPullingTime="4.5033649s" totalInitContainerRuntime="489.97828ms" isStatefulPod=false podCreationTimestamp="2026-11-02 22:56:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:56:56.477315669 +0000 UTC m=+265.445847455" watchObservedRunningTime="2026-11-02 22:56:56.478858871 +0000 UTC m=+265.447390656" Nov 02 22:57:00.810654 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:00.810619 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jcp84"] Nov 02 22:57:00.817012 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:00.816990 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jcp84" Nov 02 22:57:00.833471 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:00.833443 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zqs9\" (UniqueName: \"kubernetes.io/projected/408cfb03-2c66-47d9-862d-73e34752508e-kube-api-access-4zqs9\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-jcp84\" (UID: \"408cfb03-2c66-47d9-862d-73e34752508e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jcp84" Nov 02 22:57:00.833582 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:00.833524 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/408cfb03-2c66-47d9-862d-73e34752508e-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-jcp84\" (UID: \"408cfb03-2c66-47d9-862d-73e34752508e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jcp84" Nov 02 22:57:00.865606 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:00.865573 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jcp84"] Nov 02 22:57:00.911240 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:00.911207 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:57:00.911390 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:00.911252 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 02 22:57:00.911390 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:00.911374 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-lsvxd\"" Nov 02 22:57:00.934615 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:00.934584 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zqs9\" (UniqueName: \"kubernetes.io/projected/408cfb03-2c66-47d9-862d-73e34752508e-kube-api-access-4zqs9\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-jcp84\" (UID: \"408cfb03-2c66-47d9-862d-73e34752508e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jcp84" Nov 02 22:57:00.934707 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:00.934624 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/408cfb03-2c66-47d9-862d-73e34752508e-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-jcp84\" (UID: \"408cfb03-2c66-47d9-862d-73e34752508e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jcp84" Nov 02 22:57:00.935022 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:00.935006 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/408cfb03-2c66-47d9-862d-73e34752508e-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-jcp84\" (UID: \"408cfb03-2c66-47d9-862d-73e34752508e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jcp84" Nov 02 22:57:00.989382 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:00.989347 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zqs9\" (UniqueName: \"kubernetes.io/projected/408cfb03-2c66-47d9-862d-73e34752508e-kube-api-access-4zqs9\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-jcp84\" (UID: \"408cfb03-2c66-47d9-862d-73e34752508e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jcp84" Nov 02 22:57:01.128704 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:01.128598 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jcp84" Nov 02 22:57:01.610648 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:01.610623 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jcp84"] Nov 02 22:57:01.613069 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:57:01.613031 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod408cfb03_2c66_47d9_862d_73e34752508e.slice/crio-673ecf2bc0daa04b21fff8a6a988fa4d0966df0a592fbedfbbbf61fa83c3c748 WatchSource:0}: Error finding container 673ecf2bc0daa04b21fff8a6a988fa4d0966df0a592fbedfbbbf61fa83c3c748: Status 404 returned error can't find the container with id 673ecf2bc0daa04b21fff8a6a988fa4d0966df0a592fbedfbbbf61fa83c3c748 Nov 02 22:57:02.375902 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:02.375853 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jcp84" event={"ID":"408cfb03-2c66-47d9-862d-73e34752508e","Type":"ContainerStarted","Data":"673ecf2bc0daa04b21fff8a6a988fa4d0966df0a592fbedfbbbf61fa83c3c748"} Nov 02 22:57:03.173060 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:03.173029 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-5rw74"] Nov 02 22:57:03.177703 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:03.177682 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-5rw74" Nov 02 22:57:03.255351 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:03.255311 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g559g\" (UniqueName: \"kubernetes.io/projected/a90148ca-bc21-4a39-8be9-4cff6cc9b34c-kube-api-access-g559g\") pod \"authorino-operator-7b7bb4bc77-5rw74\" (UID: \"a90148ca-bc21-4a39-8be9-4cff6cc9b34c\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-5rw74" Nov 02 22:57:03.275290 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:03.275261 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 22:57:03.275290 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:03.275261 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-f6l6f\"" Nov 02 22:57:03.275504 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:03.275447 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 22:57:03.277682 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:03.277656 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-5rw74"] Nov 02 22:57:03.356568 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:03.356531 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g559g\" (UniqueName: \"kubernetes.io/projected/a90148ca-bc21-4a39-8be9-4cff6cc9b34c-kube-api-access-g559g\") pod \"authorino-operator-7b7bb4bc77-5rw74\" (UID: \"a90148ca-bc21-4a39-8be9-4cff6cc9b34c\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-5rw74" Nov 02 22:57:03.411003 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:03.410974 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g559g\" (UniqueName: \"kubernetes.io/projected/a90148ca-bc21-4a39-8be9-4cff6cc9b34c-kube-api-access-g559g\") pod \"authorino-operator-7b7bb4bc77-5rw74\" (UID: \"a90148ca-bc21-4a39-8be9-4cff6cc9b34c\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-5rw74" Nov 02 22:57:03.490345 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:03.490304 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-5rw74" Nov 02 22:57:03.901220 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:03.901088 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-5rw74"] Nov 02 22:57:03.903973 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:57:03.903938 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda90148ca_bc21_4a39_8be9_4cff6cc9b34c.slice/crio-755164db7e881a75c6f8172b3261c4375522412876c3ae8051602f52cc11e4e3 WatchSource:0}: Error finding container 755164db7e881a75c6f8172b3261c4375522412876c3ae8051602f52cc11e4e3: Status 404 returned error can't find the container with id 755164db7e881a75c6f8172b3261c4375522412876c3ae8051602f52cc11e4e3 Nov 02 22:57:04.386670 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:04.386636 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-5rw74" event={"ID":"a90148ca-bc21-4a39-8be9-4cff6cc9b34c","Type":"ContainerStarted","Data":"755164db7e881a75c6f8172b3261c4375522412876c3ae8051602f52cc11e4e3"} Nov 02 22:57:06.397106 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:06.397068 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jcp84" event={"ID":"408cfb03-2c66-47d9-862d-73e34752508e","Type":"ContainerStarted","Data":"f70e01f26520ae389d794185527c6d06c1957c593c9791302713ca95f3d91b26"} Nov 02 22:57:06.604045 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:06.603991 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jcp84" podStartSLOduration=2.641899343 podStartE2EDuration="6.603975128s" totalImagesPullingTime="3.962075785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:57:06.602538367 +0000 UTC m=+275.571070190" watchObservedRunningTime="2026-11-02 22:57:06.603975128 +0000 UTC m=+275.572506912" Nov 02 22:57:07.356293 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:07.356260 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-s7pdf" Nov 02 22:57:08.408748 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:08.408711 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-5rw74" event={"ID":"a90148ca-bc21-4a39-8be9-4cff6cc9b34c","Type":"ContainerStarted","Data":"bae5073f4e61f05c2a1704bb26f4f650ccce67861e631706751ce72c4bfbdb2a"} Nov 02 22:57:08.409268 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:08.408876 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-5rw74" Nov 02 22:57:08.530150 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:08.530046 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-5rw74" podStartSLOduration=1.231514499 podStartE2EDuration="5.530030255s" totalImagesPullingTime="4.298515756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:57:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:57:08.529451652 +0000 UTC m=+277.497983436" watchObservedRunningTime="2026-11-02 22:57:08.530030255 +0000 UTC m=+277.498562039" Nov 02 22:57:09.882856 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:57:09.882815 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 22:57:12.701764 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:12.701732 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-6ff5j"] Nov 02 22:57:12.704109 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:12.704092 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6ff5j" Nov 02 22:57:12.740194 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:12.740161 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhjg6\" (UniqueName: \"kubernetes.io/projected/0d983ba6-e119-4cbc-98d2-3dc824673324-kube-api-access-bhjg6\") pod \"dns-operator-controller-manager-8557cbdb84-6ff5j\" (UID: \"0d983ba6-e119-4cbc-98d2-3dc824673324\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6ff5j" Nov 02 22:57:12.754953 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:12.754925 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 02 22:57:12.802432 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:12.802403 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-5dwcz\"" Nov 02 22:57:12.803973 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:12.803953 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-6ff5j"] Nov 02 22:57:12.841073 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:12.841046 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhjg6\" (UniqueName: \"kubernetes.io/projected/0d983ba6-e119-4cbc-98d2-3dc824673324-kube-api-access-bhjg6\") pod \"dns-operator-controller-manager-8557cbdb84-6ff5j\" (UID: \"0d983ba6-e119-4cbc-98d2-3dc824673324\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6ff5j" Nov 02 22:57:12.905441 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:12.905403 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhjg6\" (UniqueName: \"kubernetes.io/projected/0d983ba6-e119-4cbc-98d2-3dc824673324-kube-api-access-bhjg6\") pod \"dns-operator-controller-manager-8557cbdb84-6ff5j\" (UID: \"0d983ba6-e119-4cbc-98d2-3dc824673324\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6ff5j" Nov 02 22:57:13.015555 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:13.015472 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6ff5j" Nov 02 22:57:13.383616 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:13.383585 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-6ff5j"] Nov 02 22:57:13.386059 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:57:13.386030 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d983ba6_e119_4cbc_98d2_3dc824673324.slice/crio-ef4ce1b515d4f8312245d2450fdc938fc72079ef78d1c6603401528ba7626349 WatchSource:0}: Error finding container ef4ce1b515d4f8312245d2450fdc938fc72079ef78d1c6603401528ba7626349: Status 404 returned error can't find the container with id ef4ce1b515d4f8312245d2450fdc938fc72079ef78d1c6603401528ba7626349 Nov 02 22:57:13.430852 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:13.430813 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6ff5j" event={"ID":"0d983ba6-e119-4cbc-98d2-3dc824673324","Type":"ContainerStarted","Data":"ef4ce1b515d4f8312245d2450fdc938fc72079ef78d1c6603401528ba7626349"} Nov 02 22:57:15.946342 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:15.946311 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-4fvc5"] Nov 02 22:57:15.949558 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:15.949537 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4fvc5" Nov 02 22:57:16.048744 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:16.048711 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-q4f7f\"" Nov 02 22:57:16.050058 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:16.050036 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-4fvc5"] Nov 02 22:57:16.070600 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:16.070570 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2lml\" (UniqueName: \"kubernetes.io/projected/c7e772da-ebf0-4dca-ac74-0c8e4cea112e-kube-api-access-f2lml\") pod \"limitador-operator-controller-manager-5778bb4449-4fvc5\" (UID: \"c7e772da-ebf0-4dca-ac74-0c8e4cea112e\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4fvc5" Nov 02 22:57:16.171773 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:16.171736 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2lml\" (UniqueName: \"kubernetes.io/projected/c7e772da-ebf0-4dca-ac74-0c8e4cea112e-kube-api-access-f2lml\") pod \"limitador-operator-controller-manager-5778bb4449-4fvc5\" (UID: \"c7e772da-ebf0-4dca-ac74-0c8e4cea112e\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4fvc5" Nov 02 22:57:16.228709 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:16.228638 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2lml\" (UniqueName: \"kubernetes.io/projected/c7e772da-ebf0-4dca-ac74-0c8e4cea112e-kube-api-access-f2lml\") pod \"limitador-operator-controller-manager-5778bb4449-4fvc5\" (UID: \"c7e772da-ebf0-4dca-ac74-0c8e4cea112e\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4fvc5" Nov 02 22:57:16.262164 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:16.262135 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4fvc5" Nov 02 22:57:16.624495 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:16.624470 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-4fvc5"] Nov 02 22:57:16.626697 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:57:16.626672 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7e772da_ebf0_4dca_ac74_0c8e4cea112e.slice/crio-8db676254f5cfbfe2d22d2650328dc53c657be147527812ad3a2b7137f56f47a WatchSource:0}: Error finding container 8db676254f5cfbfe2d22d2650328dc53c657be147527812ad3a2b7137f56f47a: Status 404 returned error can't find the container with id 8db676254f5cfbfe2d22d2650328dc53c657be147527812ad3a2b7137f56f47a Nov 02 22:57:17.450426 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:17.450380 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4fvc5" event={"ID":"c7e772da-ebf0-4dca-ac74-0c8e4cea112e","Type":"ContainerStarted","Data":"8db676254f5cfbfe2d22d2650328dc53c657be147527812ad3a2b7137f56f47a"} Nov 02 22:57:18.456207 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:18.456175 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6ff5j" event={"ID":"0d983ba6-e119-4cbc-98d2-3dc824673324","Type":"ContainerStarted","Data":"b8bfad25a0cf84d4547c8e5044517d3a39d03c3b63d1c8bbadfe1f68d9cd0a4c"} Nov 02 22:57:18.456589 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:18.456274 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6ff5j" Nov 02 22:57:18.613908 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:18.613851 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6ff5j" podStartSLOduration=2.095738368 podStartE2EDuration="6.613837144s" totalImagesPullingTime="4.518098776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:57:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:57:18.612947591 +0000 UTC m=+287.581479374" watchObservedRunningTime="2026-11-02 22:57:18.613837144 +0000 UTC m=+287.582368919" Nov 02 22:57:19.416125 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:19.416092 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-5rw74" Nov 02 22:57:20.466820 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:20.466787 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4fvc5" event={"ID":"c7e772da-ebf0-4dca-ac74-0c8e4cea112e","Type":"ContainerStarted","Data":"42bc09ab4e429b269ded943d069b73d58c49ed2055d3687a5ad6a845b1d9dfa5"} Nov 02 22:57:20.467359 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:20.466942 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4fvc5" Nov 02 22:57:20.579409 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:20.579353 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4fvc5" podStartSLOduration=2.657879552 podStartE2EDuration="5.579336328s" totalImagesPullingTime="2.921456776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:57:20.578630541 +0000 UTC m=+289.547162327" watchObservedRunningTime="2026-11-02 22:57:20.579336328 +0000 UTC m=+289.547868112" Nov 02 22:57:27.542352 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:27.542313 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft527v"] Nov 02 22:57:27.548055 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:27.548034 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft527v" Nov 02 22:57:27.616822 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:27.616793 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft527v"] Nov 02 22:57:27.662324 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:27.662290 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-6dtll\"" Nov 02 22:57:27.676231 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:27.676199 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65a56e85-82db-48c6-8060-3279887a6e89-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft527v\" (UID: \"65a56e85-82db-48c6-8060-3279887a6e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft527v" Nov 02 22:57:27.676231 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:27.676235 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a56e85-82db-48c6-8060-3279887a6e89-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft527v\" (UID: \"65a56e85-82db-48c6-8060-3279887a6e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft527v" Nov 02 22:57:27.676437 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:27.676251 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65a56e85-82db-48c6-8060-3279887a6e89-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft527v\" (UID: \"65a56e85-82db-48c6-8060-3279887a6e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft527v" Nov 02 22:57:27.676437 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:27.676270 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65a56e85-82db-48c6-8060-3279887a6e89-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft527v\" (UID: \"65a56e85-82db-48c6-8060-3279887a6e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft527v" Nov 02 22:57:27.676437 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:27.676336 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvtmj\" (UniqueName: \"kubernetes.io/projected/65a56e85-82db-48c6-8060-3279887a6e89-kube-api-access-zvtmj\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft527v\" (UID: \"65a56e85-82db-48c6-8060-3279887a6e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft527v" Nov 02 22:57:27.676532 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:27.676508 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65a56e85-82db-48c6-8060-3279887a6e89-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft527v\" (UID: \"65a56e85-82db-48c6-8060-3279887a6e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft527v" Nov 02 22:57:27.676565 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:27.676540 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a56e85-82db-48c6-8060-3279887a6e89-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft527v\" (UID: \"65a56e85-82db-48c6-8060-3279887a6e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft527v" Nov 02 22:57:27.676565 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:27.676562 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65a56e85-82db-48c6-8060-3279887a6e89-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft527v\" (UID: \"65a56e85-82db-48c6-8060-3279887a6e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft527v" Nov 02 22:57:27.676674 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:27.676661 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65a56e85-82db-48c6-8060-3279887a6e89-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft527v\" (UID: \"65a56e85-82db-48c6-8060-3279887a6e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft527v" Nov 02 22:57:27.777174 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:27.777134 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65a56e85-82db-48c6-8060-3279887a6e89-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft527v\" (UID: \"65a56e85-82db-48c6-8060-3279887a6e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft527v" Nov 02 22:57:27.777406 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:27.777183 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a56e85-82db-48c6-8060-3279887a6e89-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft527v\" (UID: \"65a56e85-82db-48c6-8060-3279887a6e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft527v" Nov 02 22:57:27.777406 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:27.777218 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65a56e85-82db-48c6-8060-3279887a6e89-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft527v\" (UID: \"65a56e85-82db-48c6-8060-3279887a6e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft527v" Nov 02 22:57:27.777406 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:27.777266 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65a56e85-82db-48c6-8060-3279887a6e89-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft527v\" (UID: \"65a56e85-82db-48c6-8060-3279887a6e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft527v" Nov 02 22:57:27.777553 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:27.777412 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65a56e85-82db-48c6-8060-3279887a6e89-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft527v\" (UID: \"65a56e85-82db-48c6-8060-3279887a6e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft527v" Nov 02 22:57:27.777553 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:27.777454 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a56e85-82db-48c6-8060-3279887a6e89-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft527v\" (UID: \"65a56e85-82db-48c6-8060-3279887a6e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft527v" Nov 02 22:57:27.777657 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:27.777584 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65a56e85-82db-48c6-8060-3279887a6e89-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft527v\" (UID: \"65a56e85-82db-48c6-8060-3279887a6e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft527v" Nov 02 22:57:27.777657 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:27.777627 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65a56e85-82db-48c6-8060-3279887a6e89-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft527v\" (UID: \"65a56e85-82db-48c6-8060-3279887a6e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft527v" Nov 02 22:57:27.777657 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:27.777628 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65a56e85-82db-48c6-8060-3279887a6e89-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft527v\" (UID: \"65a56e85-82db-48c6-8060-3279887a6e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft527v" Nov 02 22:57:27.777805 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:27.777655 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvtmj\" (UniqueName: \"kubernetes.io/projected/65a56e85-82db-48c6-8060-3279887a6e89-kube-api-access-zvtmj\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft527v\" (UID: \"65a56e85-82db-48c6-8060-3279887a6e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft527v" Nov 02 22:57:27.777805 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:27.777737 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a56e85-82db-48c6-8060-3279887a6e89-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft527v\" (UID: \"65a56e85-82db-48c6-8060-3279887a6e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft527v" Nov 02 22:57:27.777805 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:27.777791 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a56e85-82db-48c6-8060-3279887a6e89-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft527v\" (UID: \"65a56e85-82db-48c6-8060-3279887a6e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft527v" Nov 02 22:57:27.777949 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:27.777927 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65a56e85-82db-48c6-8060-3279887a6e89-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft527v\" (UID: \"65a56e85-82db-48c6-8060-3279887a6e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft527v" Nov 02 22:57:27.778356 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:27.778327 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65a56e85-82db-48c6-8060-3279887a6e89-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft527v\" (UID: \"65a56e85-82db-48c6-8060-3279887a6e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft527v" Nov 02 22:57:27.779844 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:27.779822 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65a56e85-82db-48c6-8060-3279887a6e89-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft527v\" (UID: \"65a56e85-82db-48c6-8060-3279887a6e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft527v" Nov 02 22:57:27.780119 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:27.780101 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65a56e85-82db-48c6-8060-3279887a6e89-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft527v\" (UID: \"65a56e85-82db-48c6-8060-3279887a6e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft527v" Nov 02 22:57:27.834423 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:27.834348 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65a56e85-82db-48c6-8060-3279887a6e89-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft527v\" (UID: \"65a56e85-82db-48c6-8060-3279887a6e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft527v" Nov 02 22:57:27.834564 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:27.834427 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvtmj\" (UniqueName: \"kubernetes.io/projected/65a56e85-82db-48c6-8060-3279887a6e89-kube-api-access-zvtmj\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898ft527v\" (UID: \"65a56e85-82db-48c6-8060-3279887a6e89\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft527v" Nov 02 22:57:27.859249 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:27.859223 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft527v" Nov 02 22:57:28.277015 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:28.276990 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft527v"] Nov 02 22:57:28.279463 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:57:28.279424 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65a56e85_82db_48c6_8060_3279887a6e89.slice/crio-b1a4368da7322dfba2f0443a4034ed5092917861db919f72e95ffa735ff60f38 WatchSource:0}: Error finding container b1a4368da7322dfba2f0443a4034ed5092917861db919f72e95ffa735ff60f38: Status 404 returned error can't find the container with id b1a4368da7322dfba2f0443a4034ed5092917861db919f72e95ffa735ff60f38 Nov 02 22:57:28.281465 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:28.281424 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:57:28.281542 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:28.281516 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:57:28.281580 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:28.281548 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:57:28.415520 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:28.415485 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-9djtl"] Nov 02 22:57:28.419064 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:28.419042 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-9djtl" Nov 02 22:57:28.471872 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:28.471834 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-9djtl"] Nov 02 22:57:28.506626 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:28.506589 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft527v" event={"ID":"65a56e85-82db-48c6-8060-3279887a6e89","Type":"ContainerStarted","Data":"0cf42200dd5e452c88c5523e9d85a06e49c874569121dbd7f1961cd4688a2ded"} Nov 02 22:57:28.506791 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:28.506658 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft527v" event={"ID":"65a56e85-82db-48c6-8060-3279887a6e89","Type":"ContainerStarted","Data":"b1a4368da7322dfba2f0443a4034ed5092917861db919f72e95ffa735ff60f38"} Nov 02 22:57:28.519325 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:28.519296 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-p54bz\"" Nov 02 22:57:28.584515 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:28.584435 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vgbf\" (UniqueName: \"kubernetes.io/projected/b732f7d2-e75a-4534-98ea-82afc9076de0-kube-api-access-6vgbf\") pod \"authorino-57f7f8659f-9djtl\" (UID: \"b732f7d2-e75a-4534-98ea-82afc9076de0\") " pod="kuadrant-system/authorino-57f7f8659f-9djtl" Nov 02 22:57:28.685368 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:28.685331 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vgbf\" (UniqueName: \"kubernetes.io/projected/b732f7d2-e75a-4534-98ea-82afc9076de0-kube-api-access-6vgbf\") pod \"authorino-57f7f8659f-9djtl\" (UID: \"b732f7d2-e75a-4534-98ea-82afc9076de0\") " pod="kuadrant-system/authorino-57f7f8659f-9djtl" Nov 02 22:57:28.739624 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:28.739582 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vgbf\" (UniqueName: \"kubernetes.io/projected/b732f7d2-e75a-4534-98ea-82afc9076de0-kube-api-access-6vgbf\") pod \"authorino-57f7f8659f-9djtl\" (UID: \"b732f7d2-e75a-4534-98ea-82afc9076de0\") " pod="kuadrant-system/authorino-57f7f8659f-9djtl" Nov 02 22:57:28.746389 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:28.746372 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-9djtl" Nov 02 22:57:28.859816 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:28.859744 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft527v" Nov 02 22:57:29.110560 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:29.110453 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft527v" podStartSLOduration=2.110437174 podStartE2EDuration="2.110437174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:57:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:57:28.642359515 +0000 UTC m=+297.610891302" watchObservedRunningTime="2026-11-02 22:57:29.110437174 +0000 UTC m=+298.078968958" Nov 02 22:57:29.111031 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:29.110833 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-9djtl"] Nov 02 22:57:29.113564 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:57:29.113536 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb732f7d2_e75a_4534_98ea_82afc9076de0.slice/crio-5bc5d35755179d6b94876ef03cc6d4fe08682b775a2c135b134e16a88dbc6ef0 WatchSource:0}: Error finding container 5bc5d35755179d6b94876ef03cc6d4fe08682b775a2c135b134e16a88dbc6ef0: Status 404 returned error can't find the container with id 5bc5d35755179d6b94876ef03cc6d4fe08682b775a2c135b134e16a88dbc6ef0 Nov 02 22:57:29.234215 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:29.234180 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-dlhsg"] Nov 02 22:57:29.237130 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:29.237112 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-dlhsg" Nov 02 22:57:29.289012 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:29.288875 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 02 22:57:29.289012 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:29.288959 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 02 22:57:29.289012 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:29.288992 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-rp69l\"" Nov 02 22:57:29.289883 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:29.289859 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-dlhsg"] Nov 02 22:57:29.393021 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:29.392943 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5010147d-f932-4137-bfd8-5d7fc3308309-bound-sa-token\") pod \"cert-manager-597b95cb84-dlhsg\" (UID: \"5010147d-f932-4137-bfd8-5d7fc3308309\") " pod="cert-manager/cert-manager-597b95cb84-dlhsg" Nov 02 22:57:29.393169 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:29.393017 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8shxm\" (UniqueName: \"kubernetes.io/projected/5010147d-f932-4137-bfd8-5d7fc3308309-kube-api-access-8shxm\") pod \"cert-manager-597b95cb84-dlhsg\" (UID: \"5010147d-f932-4137-bfd8-5d7fc3308309\") " pod="cert-manager/cert-manager-597b95cb84-dlhsg" Nov 02 22:57:29.463023 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:29.462992 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6ff5j" Nov 02 22:57:29.494352 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:29.494322 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5010147d-f932-4137-bfd8-5d7fc3308309-bound-sa-token\") pod \"cert-manager-597b95cb84-dlhsg\" (UID: \"5010147d-f932-4137-bfd8-5d7fc3308309\") " pod="cert-manager/cert-manager-597b95cb84-dlhsg" Nov 02 22:57:29.494495 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:29.494374 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8shxm\" (UniqueName: \"kubernetes.io/projected/5010147d-f932-4137-bfd8-5d7fc3308309-kube-api-access-8shxm\") pod \"cert-manager-597b95cb84-dlhsg\" (UID: \"5010147d-f932-4137-bfd8-5d7fc3308309\") " pod="cert-manager/cert-manager-597b95cb84-dlhsg" Nov 02 22:57:29.512839 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:29.512800 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-9djtl" event={"ID":"b732f7d2-e75a-4534-98ea-82afc9076de0","Type":"ContainerStarted","Data":"5bc5d35755179d6b94876ef03cc6d4fe08682b775a2c135b134e16a88dbc6ef0"} Nov 02 22:57:29.550849 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:29.550816 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5010147d-f932-4137-bfd8-5d7fc3308309-bound-sa-token\") pod \"cert-manager-597b95cb84-dlhsg\" (UID: \"5010147d-f932-4137-bfd8-5d7fc3308309\") " pod="cert-manager/cert-manager-597b95cb84-dlhsg" Nov 02 22:57:29.551010 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:29.550951 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8shxm\" (UniqueName: \"kubernetes.io/projected/5010147d-f932-4137-bfd8-5d7fc3308309-kube-api-access-8shxm\") pod \"cert-manager-597b95cb84-dlhsg\" (UID: \"5010147d-f932-4137-bfd8-5d7fc3308309\") " pod="cert-manager/cert-manager-597b95cb84-dlhsg" Nov 02 22:57:29.848058 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:29.848023 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-dlhsg" Nov 02 22:57:29.865531 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:29.865506 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft527v" Nov 02 22:57:30.215659 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:30.215635 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-dlhsg"] Nov 02 22:57:30.217460 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:57:30.217432 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5010147d_f932_4137_bfd8_5d7fc3308309.slice/crio-b55bea376fe1303baa948aa565443cc72b0a0d6939ef000c3f58df35ac1f2ae5 WatchSource:0}: Error finding container b55bea376fe1303baa948aa565443cc72b0a0d6939ef000c3f58df35ac1f2ae5: Status 404 returned error can't find the container with id b55bea376fe1303baa948aa565443cc72b0a0d6939ef000c3f58df35ac1f2ae5 Nov 02 22:57:30.519163 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:30.519083 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-dlhsg" event={"ID":"5010147d-f932-4137-bfd8-5d7fc3308309","Type":"ContainerStarted","Data":"b55bea376fe1303baa948aa565443cc72b0a0d6939ef000c3f58df35ac1f2ae5"} Nov 02 22:57:30.519320 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:30.519278 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft527v" Nov 02 22:57:30.520331 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:30.520311 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898ft527v" Nov 02 22:57:31.186193 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:31.186162 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 02 22:57:31.189197 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:31.189173 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 02 22:57:31.473292 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:31.473213 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4fvc5" Nov 02 22:57:31.521191 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:31.521162 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-lmk64"] Nov 02 22:57:31.524466 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:31.524442 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-lmk64" Nov 02 22:57:31.575455 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:31.575426 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-lmk64"] Nov 02 22:57:31.613120 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:31.613090 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8478\" (UniqueName: \"kubernetes.io/projected/f9575964-4fe1-4130-b639-ceb989948972-kube-api-access-r8478\") pod \"authorino-56c6c9ddd5-lmk64\" (UID: \"f9575964-4fe1-4130-b639-ceb989948972\") " pod="kuadrant-system/authorino-56c6c9ddd5-lmk64" Nov 02 22:57:31.713879 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:31.713838 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8478\" (UniqueName: \"kubernetes.io/projected/f9575964-4fe1-4130-b639-ceb989948972-kube-api-access-r8478\") pod \"authorino-56c6c9ddd5-lmk64\" (UID: \"f9575964-4fe1-4130-b639-ceb989948972\") " pod="kuadrant-system/authorino-56c6c9ddd5-lmk64" Nov 02 22:57:31.771334 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:31.771260 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8478\" (UniqueName: \"kubernetes.io/projected/f9575964-4fe1-4130-b639-ceb989948972-kube-api-access-r8478\") pod \"authorino-56c6c9ddd5-lmk64\" (UID: \"f9575964-4fe1-4130-b639-ceb989948972\") " pod="kuadrant-system/authorino-56c6c9ddd5-lmk64" Nov 02 22:57:31.836226 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:31.836195 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-lmk64" Nov 02 22:57:32.222793 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:57:32.222761 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9575964_4fe1_4130_b639_ceb989948972.slice/crio-a7a23e0b161f9a80ea8c9163aaaf413afb0c19284e59b14003ea2a7ecc01d712 WatchSource:0}: Error finding container a7a23e0b161f9a80ea8c9163aaaf413afb0c19284e59b14003ea2a7ecc01d712: Status 404 returned error can't find the container with id a7a23e0b161f9a80ea8c9163aaaf413afb0c19284e59b14003ea2a7ecc01d712 Nov 02 22:57:32.226591 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:32.226564 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-lmk64"] Nov 02 22:57:32.531524 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:32.531434 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-9djtl" event={"ID":"b732f7d2-e75a-4534-98ea-82afc9076de0","Type":"ContainerStarted","Data":"f8c185eb9bdc7a3e5fabef6fa0159c4e75d4f867877e17695c2f9936a29e80d2"} Nov 02 22:57:32.533629 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:32.533599 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-lmk64" event={"ID":"f9575964-4fe1-4130-b639-ceb989948972","Type":"ContainerStarted","Data":"a7a23e0b161f9a80ea8c9163aaaf413afb0c19284e59b14003ea2a7ecc01d712"} Nov 02 22:57:32.639831 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:32.639758 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-9djtl" podStartSLOduration=1.369124712 podStartE2EDuration="4.639743922s" totalImagesPullingTime="3.27061921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:57:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:57:32.639104444 +0000 UTC m=+301.607636228" watchObservedRunningTime="2026-11-02 22:57:32.639743922 +0000 UTC m=+301.608275708" Nov 02 22:57:34.544573 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:34.544532 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-dlhsg" event={"ID":"5010147d-f932-4137-bfd8-5d7fc3308309","Type":"ContainerStarted","Data":"7cafefbfa6b391c0e1c9b93cc3c6168c95f0a1002dabc88475f2bf2f37a9b3dd"} Nov 02 22:57:34.545886 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:34.545861 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-lmk64" event={"ID":"f9575964-4fe1-4130-b639-ceb989948972","Type":"ContainerStarted","Data":"c6b1679529f58ad5662333cf9e2ed79f6424770500d1701eab9a533675d55ea8"} Nov 02 22:57:34.657491 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:34.657435 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-dlhsg" podStartSLOduration=2.026686185 podStartE2EDuration="5.657421035s" totalImagesPullingTime="3.63073485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:57:34.655993089 +0000 UTC m=+303.624524879" watchObservedRunningTime="2026-11-02 22:57:34.657421035 +0000 UTC m=+303.625952818" Nov 02 22:57:34.762829 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:34.762753 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-lmk64" podStartSLOduration=2.354569686 podStartE2EDuration="3.762734113s" totalImagesPullingTime="1.408164427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:57:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:57:34.762421989 +0000 UTC m=+303.730953775" watchObservedRunningTime="2026-11-02 22:57:34.762734113 +0000 UTC m=+303.731265897" Nov 02 22:57:34.786380 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:34.786346 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-9djtl"] Nov 02 22:57:34.786567 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:34.786542 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-9djtl" podUID="b732f7d2-e75a-4534-98ea-82afc9076de0" containerName="authorino" containerID="cri-o://f8c185eb9bdc7a3e5fabef6fa0159c4e75d4f867877e17695c2f9936a29e80d2" gracePeriod=30 Nov 02 22:57:35.032107 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:35.032078 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-9djtl" Nov 02 22:57:35.148517 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:35.148438 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b732f7d2-e75a-4534-98ea-82afc9076de0-kube-api-access-6vgbf\" (UniqueName: \"kubernetes.io/projected/b732f7d2-e75a-4534-98ea-82afc9076de0-kube-api-access-6vgbf\") pod \"b732f7d2-e75a-4534-98ea-82afc9076de0\" (UID: \"b732f7d2-e75a-4534-98ea-82afc9076de0\") " Nov 02 22:57:35.150762 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:35.150722 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b732f7d2-e75a-4534-98ea-82afc9076de0-kube-api-access-6vgbf" pod "b732f7d2-e75a-4534-98ea-82afc9076de0" (UID: "b732f7d2-e75a-4534-98ea-82afc9076de0"). InnerVolumeSpecName "kube-api-access-6vgbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:57:35.249695 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:35.249653 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vgbf\" (UniqueName: \"kubernetes.io/projected/b732f7d2-e75a-4534-98ea-82afc9076de0-kube-api-access-6vgbf\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:57:35.551774 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:35.551735 5076 generic.go:356] "Generic (PLEG): container finished" podID="b732f7d2-e75a-4534-98ea-82afc9076de0" containerID="f8c185eb9bdc7a3e5fabef6fa0159c4e75d4f867877e17695c2f9936a29e80d2" exitCode=0 Nov 02 22:57:35.551774 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:35.551781 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-9djtl" Nov 02 22:57:35.552328 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:35.551818 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-9djtl" event={"ID":"b732f7d2-e75a-4534-98ea-82afc9076de0","Type":"ContainerDied","Data":"f8c185eb9bdc7a3e5fabef6fa0159c4e75d4f867877e17695c2f9936a29e80d2"} Nov 02 22:57:35.552328 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:35.551866 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-9djtl" event={"ID":"b732f7d2-e75a-4534-98ea-82afc9076de0","Type":"ContainerDied","Data":"5bc5d35755179d6b94876ef03cc6d4fe08682b775a2c135b134e16a88dbc6ef0"} Nov 02 22:57:35.552328 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:35.551911 5076 scope.go:122] "RemoveContainer" containerID="f8c185eb9bdc7a3e5fabef6fa0159c4e75d4f867877e17695c2f9936a29e80d2" Nov 02 22:57:35.561317 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:35.561300 5076 scope.go:122] "RemoveContainer" containerID="f8c185eb9bdc7a3e5fabef6fa0159c4e75d4f867877e17695c2f9936a29e80d2" Nov 02 22:57:35.561568 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:57:35.561552 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8c185eb9bdc7a3e5fabef6fa0159c4e75d4f867877e17695c2f9936a29e80d2\": container with ID starting with f8c185eb9bdc7a3e5fabef6fa0159c4e75d4f867877e17695c2f9936a29e80d2 not found: ID does not exist" containerID="f8c185eb9bdc7a3e5fabef6fa0159c4e75d4f867877e17695c2f9936a29e80d2" Nov 02 22:57:35.561610 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:35.561578 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8c185eb9bdc7a3e5fabef6fa0159c4e75d4f867877e17695c2f9936a29e80d2"} err="failed to get container status \"f8c185eb9bdc7a3e5fabef6fa0159c4e75d4f867877e17695c2f9936a29e80d2\": rpc error: code = NotFound desc = could not find container \"f8c185eb9bdc7a3e5fabef6fa0159c4e75d4f867877e17695c2f9936a29e80d2\": container with ID starting with f8c185eb9bdc7a3e5fabef6fa0159c4e75d4f867877e17695c2f9936a29e80d2 not found: ID does not exist" Nov 02 22:57:35.709025 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:35.708988 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-9djtl"] Nov 02 22:57:35.714391 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:35.714366 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-9djtl"] Nov 02 22:57:37.146019 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:37.145986 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b732f7d2-e75a-4534-98ea-82afc9076de0" path="/var/lib/kubelet/pods/b732f7d2-e75a-4534-98ea-82afc9076de0/volumes" Nov 02 22:57:39.882016 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:57:39.881979 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 22:57:42.555482 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:42.555409 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ll4vj"] Nov 02 22:57:42.555956 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:42.555845 5076 cpu_manager.go:414] "containerMap: removing container" podUID="b732f7d2-e75a-4534-98ea-82afc9076de0" containerName="authorino" Nov 02 22:57:42.555956 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:42.555859 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b732f7d2-e75a-4534-98ea-82afc9076de0" containerName="authorino" Nov 02 22:57:42.555956 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:42.555942 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="b732f7d2-e75a-4534-98ea-82afc9076de0" containerName="authorino" Nov 02 22:57:42.558727 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:42.558708 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-ll4vj" Nov 02 22:57:42.611175 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:42.611146 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ll4vj"] Nov 02 22:57:42.656538 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:42.656508 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 02 22:57:42.656538 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:42.656521 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-clk27\"" Nov 02 22:57:42.716540 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:42.716505 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3a48bfca-9d22-4158-9f4d-3db27e41fbf3-config-file\") pod \"limitador-limitador-74dfcfb894-ll4vj\" (UID: \"3a48bfca-9d22-4158-9f4d-3db27e41fbf3\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-ll4vj" Nov 02 22:57:42.716712 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:42.716548 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-799hq\" (UniqueName: \"kubernetes.io/projected/3a48bfca-9d22-4158-9f4d-3db27e41fbf3-kube-api-access-799hq\") pod \"limitador-limitador-74dfcfb894-ll4vj\" (UID: \"3a48bfca-9d22-4158-9f4d-3db27e41fbf3\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-ll4vj" Nov 02 22:57:42.792631 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:42.792598 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ll4vj"] Nov 02 22:57:42.817592 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:42.817517 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3a48bfca-9d22-4158-9f4d-3db27e41fbf3-config-file\") pod \"limitador-limitador-74dfcfb894-ll4vj\" (UID: \"3a48bfca-9d22-4158-9f4d-3db27e41fbf3\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-ll4vj" Nov 02 22:57:42.817592 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:42.817558 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-799hq\" (UniqueName: \"kubernetes.io/projected/3a48bfca-9d22-4158-9f4d-3db27e41fbf3-kube-api-access-799hq\") pod \"limitador-limitador-74dfcfb894-ll4vj\" (UID: \"3a48bfca-9d22-4158-9f4d-3db27e41fbf3\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-ll4vj" Nov 02 22:57:42.818214 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:42.818195 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3a48bfca-9d22-4158-9f4d-3db27e41fbf3-config-file\") pod \"limitador-limitador-74dfcfb894-ll4vj\" (UID: \"3a48bfca-9d22-4158-9f4d-3db27e41fbf3\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-ll4vj" Nov 02 22:57:42.871737 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:42.871698 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-799hq\" (UniqueName: \"kubernetes.io/projected/3a48bfca-9d22-4158-9f4d-3db27e41fbf3-kube-api-access-799hq\") pod \"limitador-limitador-74dfcfb894-ll4vj\" (UID: \"3a48bfca-9d22-4158-9f4d-3db27e41fbf3\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-ll4vj" Nov 02 22:57:43.171595 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:43.171516 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-ll4vj" Nov 02 22:57:43.536778 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:43.536750 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ll4vj"] Nov 02 22:57:43.539717 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:57:43.539677 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a48bfca_9d22_4158_9f4d_3db27e41fbf3.slice/crio-482fe04391e69d11787a6d9fb38d3cf27a037884d0856cfa357c116fe4adb40f WatchSource:0}: Error finding container 482fe04391e69d11787a6d9fb38d3cf27a037884d0856cfa357c116fe4adb40f: Status 404 returned error can't find the container with id 482fe04391e69d11787a6d9fb38d3cf27a037884d0856cfa357c116fe4adb40f Nov 02 22:57:43.541972 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:43.541953 5076 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:57:43.595318 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:43.595281 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-ll4vj" event={"ID":"3a48bfca-9d22-4158-9f4d-3db27e41fbf3","Type":"ContainerStarted","Data":"482fe04391e69d11787a6d9fb38d3cf27a037884d0856cfa357c116fe4adb40f"} Nov 02 22:57:46.611426 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:46.611383 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-ll4vj" event={"ID":"3a48bfca-9d22-4158-9f4d-3db27e41fbf3","Type":"ContainerStarted","Data":"07c242188afb324ba8802a4d371cf606755f88276cb94b6aee2b8babd40e4101"} Nov 02 22:57:46.611962 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:46.611533 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-ll4vj" Nov 02 22:57:46.723001 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:46.722943 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-ll4vj" podStartSLOduration=2.358650778 podStartE2EDuration="4.722928413s" totalImagesPullingTime="2.364277635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:57:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:57:46.72139547 +0000 UTC m=+315.689927255" watchObservedRunningTime="2026-11-02 22:57:46.722928413 +0000 UTC m=+315.691460197" Nov 02 22:57:52.494319 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:52.494285 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-hc5jd"] Nov 02 22:57:52.498435 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:52.498418 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hc5jd" Nov 02 22:57:52.548451 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:52.548423 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 22:57:52.548614 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:52.548458 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-brwjj\"" Nov 02 22:57:52.596280 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:52.596250 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 22:57:52.596623 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:52.596598 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjfqj\" (UniqueName: \"kubernetes.io/projected/5c78102f-a925-4cb0-bc69-971c0d248e4e-kube-api-access-xjfqj\") pod \"kserve-module-controller-manager-7566cf5fc8-hc5jd\" (UID: \"5c78102f-a925-4cb0-bc69-971c0d248e4e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hc5jd" Nov 02 22:57:52.596664 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:52.596641 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5c78102f-a925-4cb0-bc69-971c0d248e4e-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-hc5jd\" (UID: \"5c78102f-a925-4cb0-bc69-971c0d248e4e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hc5jd" Nov 02 22:57:52.596821 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:52.596795 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c78102f-a925-4cb0-bc69-971c0d248e4e-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-hc5jd\" (UID: \"5c78102f-a925-4cb0-bc69-971c0d248e4e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hc5jd" Nov 02 22:57:52.596873 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:52.596858 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5c78102f-a925-4cb0-bc69-971c0d248e4e-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-hc5jd\" (UID: \"5c78102f-a925-4cb0-bc69-971c0d248e4e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hc5jd" Nov 02 22:57:52.692211 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:52.692165 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 02 22:57:52.693846 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:52.693827 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-hc5jd"] Nov 02 22:57:52.697726 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:52.697703 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c78102f-a925-4cb0-bc69-971c0d248e4e-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-hc5jd\" (UID: \"5c78102f-a925-4cb0-bc69-971c0d248e4e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hc5jd" Nov 02 22:57:52.697815 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:52.697743 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5c78102f-a925-4cb0-bc69-971c0d248e4e-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-hc5jd\" (UID: \"5c78102f-a925-4cb0-bc69-971c0d248e4e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hc5jd" Nov 02 22:57:52.697815 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:52.697777 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjfqj\" (UniqueName: \"kubernetes.io/projected/5c78102f-a925-4cb0-bc69-971c0d248e4e-kube-api-access-xjfqj\") pod \"kserve-module-controller-manager-7566cf5fc8-hc5jd\" (UID: \"5c78102f-a925-4cb0-bc69-971c0d248e4e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hc5jd" Nov 02 22:57:52.697815 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:52.697805 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5c78102f-a925-4cb0-bc69-971c0d248e4e-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-hc5jd\" (UID: \"5c78102f-a925-4cb0-bc69-971c0d248e4e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hc5jd" Nov 02 22:57:52.698113 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:52.698093 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c78102f-a925-4cb0-bc69-971c0d248e4e-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-hc5jd\" (UID: \"5c78102f-a925-4cb0-bc69-971c0d248e4e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hc5jd" Nov 02 22:57:52.698179 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:52.698133 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5c78102f-a925-4cb0-bc69-971c0d248e4e-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-hc5jd\" (UID: \"5c78102f-a925-4cb0-bc69-971c0d248e4e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hc5jd" Nov 02 22:57:52.698631 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:52.698611 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5c78102f-a925-4cb0-bc69-971c0d248e4e-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-hc5jd\" (UID: \"5c78102f-a925-4cb0-bc69-971c0d248e4e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hc5jd" Nov 02 22:57:52.752304 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:52.752226 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjfqj\" (UniqueName: \"kubernetes.io/projected/5c78102f-a925-4cb0-bc69-971c0d248e4e-kube-api-access-xjfqj\") pod \"kserve-module-controller-manager-7566cf5fc8-hc5jd\" (UID: \"5c78102f-a925-4cb0-bc69-971c0d248e4e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hc5jd" Nov 02 22:57:52.810263 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:52.810228 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hc5jd" Nov 02 22:57:53.218679 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:53.218655 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-hc5jd"] Nov 02 22:57:53.220443 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:57:53.220417 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c78102f_a925_4cb0_bc69_971c0d248e4e.slice/crio-0b827eeed094f5e679ff5a8306dcf2310f626531d7aec39d4a9cb914f8d8348a WatchSource:0}: Error finding container 0b827eeed094f5e679ff5a8306dcf2310f626531d7aec39d4a9cb914f8d8348a: Status 404 returned error can't find the container with id 0b827eeed094f5e679ff5a8306dcf2310f626531d7aec39d4a9cb914f8d8348a Nov 02 22:57:53.645090 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:53.645005 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hc5jd" event={"ID":"5c78102f-a925-4cb0-bc69-971c0d248e4e","Type":"ContainerStarted","Data":"0b827eeed094f5e679ff5a8306dcf2310f626531d7aec39d4a9cb914f8d8348a"} Nov 02 22:57:56.663356 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:56.663306 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hc5jd" event={"ID":"5c78102f-a925-4cb0-bc69-971c0d248e4e","Type":"ContainerStarted","Data":"aadb8ab3b25bfff75e2581b9b08f0758985b74bdd3892cdb06823eeff5f28dc7"} Nov 02 22:57:56.663765 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:56.663485 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hc5jd" Nov 02 22:57:56.822261 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:56.822196 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hc5jd" podStartSLOduration=1.620556578 podStartE2EDuration="4.82218262s" totalImagesPullingTime="3.201626042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:57:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:57:56.820284447 +0000 UTC m=+325.788816231" watchObservedRunningTime="2026-11-02 22:57:56.82218262 +0000 UTC m=+325.790714450" Nov 02 22:57:57.617841 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:57:57.617811 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-ll4vj" Nov 02 22:58:01.182602 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:01.182569 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ll4vj"] Nov 02 22:58:01.183049 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:01.182765 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-ll4vj" podUID="3a48bfca-9d22-4158-9f4d-3db27e41fbf3" containerName="limitador" containerID="cri-o://07c242188afb324ba8802a4d371cf606755f88276cb94b6aee2b8babd40e4101" gracePeriod=30 Nov 02 22:58:01.636200 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:01.636175 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-ll4vj" Nov 02 22:58:01.661333 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:01.661300 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a48bfca-9d22-4158-9f4d-3db27e41fbf3-config-file\" (UniqueName: \"kubernetes.io/configmap/3a48bfca-9d22-4158-9f4d-3db27e41fbf3-config-file\") pod \"3a48bfca-9d22-4158-9f4d-3db27e41fbf3\" (UID: \"3a48bfca-9d22-4158-9f4d-3db27e41fbf3\") " Nov 02 22:58:01.661476 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:01.661436 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a48bfca-9d22-4158-9f4d-3db27e41fbf3-kube-api-access-799hq\" (UniqueName: \"kubernetes.io/projected/3a48bfca-9d22-4158-9f4d-3db27e41fbf3-kube-api-access-799hq\") pod \"3a48bfca-9d22-4158-9f4d-3db27e41fbf3\" (UID: \"3a48bfca-9d22-4158-9f4d-3db27e41fbf3\") " Nov 02 22:58:01.661611 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:01.661589 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a48bfca-9d22-4158-9f4d-3db27e41fbf3-config-file" pod "3a48bfca-9d22-4158-9f4d-3db27e41fbf3" (UID: "3a48bfca-9d22-4158-9f4d-3db27e41fbf3"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:58:01.663398 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:01.663370 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a48bfca-9d22-4158-9f4d-3db27e41fbf3-kube-api-access-799hq" pod "3a48bfca-9d22-4158-9f4d-3db27e41fbf3" (UID: "3a48bfca-9d22-4158-9f4d-3db27e41fbf3"). InnerVolumeSpecName "kube-api-access-799hq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:58:01.688554 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:01.688519 5076 generic.go:356] "Generic (PLEG): container finished" podID="3a48bfca-9d22-4158-9f4d-3db27e41fbf3" containerID="07c242188afb324ba8802a4d371cf606755f88276cb94b6aee2b8babd40e4101" exitCode=0 Nov 02 22:58:01.688728 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:01.688563 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-ll4vj" Nov 02 22:58:01.688728 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:01.688604 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-ll4vj" event={"ID":"3a48bfca-9d22-4158-9f4d-3db27e41fbf3","Type":"ContainerDied","Data":"07c242188afb324ba8802a4d371cf606755f88276cb94b6aee2b8babd40e4101"} Nov 02 22:58:01.688728 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:01.688665 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-ll4vj" event={"ID":"3a48bfca-9d22-4158-9f4d-3db27e41fbf3","Type":"ContainerDied","Data":"482fe04391e69d11787a6d9fb38d3cf27a037884d0856cfa357c116fe4adb40f"} Nov 02 22:58:01.688728 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:01.688698 5076 scope.go:122] "RemoveContainer" containerID="07c242188afb324ba8802a4d371cf606755f88276cb94b6aee2b8babd40e4101" Nov 02 22:58:01.697473 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:01.697454 5076 scope.go:122] "RemoveContainer" containerID="07c242188afb324ba8802a4d371cf606755f88276cb94b6aee2b8babd40e4101" Nov 02 22:58:01.697735 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:58:01.697715 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07c242188afb324ba8802a4d371cf606755f88276cb94b6aee2b8babd40e4101\": container with ID starting with 07c242188afb324ba8802a4d371cf606755f88276cb94b6aee2b8babd40e4101 not found: ID does not exist" containerID="07c242188afb324ba8802a4d371cf606755f88276cb94b6aee2b8babd40e4101" Nov 02 22:58:01.697783 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:01.697743 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07c242188afb324ba8802a4d371cf606755f88276cb94b6aee2b8babd40e4101"} err="failed to get container status \"07c242188afb324ba8802a4d371cf606755f88276cb94b6aee2b8babd40e4101\": rpc error: code = NotFound desc = could not find container \"07c242188afb324ba8802a4d371cf606755f88276cb94b6aee2b8babd40e4101\": container with ID starting with 07c242188afb324ba8802a4d371cf606755f88276cb94b6aee2b8babd40e4101 not found: ID does not exist" Nov 02 22:58:01.762070 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:01.761992 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-799hq\" (UniqueName: \"kubernetes.io/projected/3a48bfca-9d22-4158-9f4d-3db27e41fbf3-kube-api-access-799hq\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:58:01.762070 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:01.762022 5076 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3a48bfca-9d22-4158-9f4d-3db27e41fbf3-config-file\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:58:01.849724 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:01.849687 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ll4vj"] Nov 02 22:58:01.852631 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:01.852602 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ll4vj"] Nov 02 22:58:03.147127 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:03.147090 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a48bfca-9d22-4158-9f4d-3db27e41fbf3" path="/var/lib/kubelet/pods/3a48bfca-9d22-4158-9f4d-3db27e41fbf3/volumes" Nov 02 22:58:08.376184 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:08.376148 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-4tbpp"] Nov 02 22:58:08.376650 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:08.376517 5076 cpu_manager.go:414] "containerMap: removing container" podUID="3a48bfca-9d22-4158-9f4d-3db27e41fbf3" containerName="limitador" Nov 02 22:58:08.376650 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:08.376527 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a48bfca-9d22-4158-9f4d-3db27e41fbf3" containerName="limitador" Nov 02 22:58:08.376650 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:08.376621 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a48bfca-9d22-4158-9f4d-3db27e41fbf3" containerName="limitador" Nov 02 22:58:08.381801 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:08.381779 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4tbpp" Nov 02 22:58:08.408298 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:08.408262 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/818b5c39-d141-455e-9ff2-ff03b4ecc69d-data\") pod \"postgres-7cdbf6dc44-4tbpp\" (UID: \"818b5c39-d141-455e-9ff2-ff03b4ecc69d\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4tbpp" Nov 02 22:58:08.408456 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:08.408389 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blvxn\" (UniqueName: \"kubernetes.io/projected/818b5c39-d141-455e-9ff2-ff03b4ecc69d-kube-api-access-blvxn\") pod \"postgres-7cdbf6dc44-4tbpp\" (UID: \"818b5c39-d141-455e-9ff2-ff03b4ecc69d\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4tbpp" Nov 02 22:58:08.432620 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:08.432587 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-4tbpp"] Nov 02 22:58:08.478267 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:08.478239 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:58:08.478425 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:08.478298 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 02 22:58:08.478425 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:08.478303 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:58:08.478425 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:08.478365 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-qhrgn\"" Nov 02 22:58:08.509089 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:08.509048 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blvxn\" (UniqueName: \"kubernetes.io/projected/818b5c39-d141-455e-9ff2-ff03b4ecc69d-kube-api-access-blvxn\") pod \"postgres-7cdbf6dc44-4tbpp\" (UID: \"818b5c39-d141-455e-9ff2-ff03b4ecc69d\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4tbpp" Nov 02 22:58:08.509257 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:08.509139 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/818b5c39-d141-455e-9ff2-ff03b4ecc69d-data\") pod \"postgres-7cdbf6dc44-4tbpp\" (UID: \"818b5c39-d141-455e-9ff2-ff03b4ecc69d\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4tbpp" Nov 02 22:58:08.509504 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:08.509483 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/818b5c39-d141-455e-9ff2-ff03b4ecc69d-data\") pod \"postgres-7cdbf6dc44-4tbpp\" (UID: \"818b5c39-d141-455e-9ff2-ff03b4ecc69d\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4tbpp" Nov 02 22:58:08.562737 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:08.562702 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blvxn\" (UniqueName: \"kubernetes.io/projected/818b5c39-d141-455e-9ff2-ff03b4ecc69d-kube-api-access-blvxn\") pod \"postgres-7cdbf6dc44-4tbpp\" (UID: \"818b5c39-d141-455e-9ff2-ff03b4ecc69d\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4tbpp" Nov 02 22:58:08.694387 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:08.694359 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4tbpp" Nov 02 22:58:09.060217 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:09.060195 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-4tbpp"] Nov 02 22:58:09.063164 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:58:09.063135 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod818b5c39_d141_455e_9ff2_ff03b4ecc69d.slice/crio-7da3a3b4637ec8056f3010f3fd0125f547ec4fefe54dced2214ac5990c0ed5e0 WatchSource:0}: Error finding container 7da3a3b4637ec8056f3010f3fd0125f547ec4fefe54dced2214ac5990c0ed5e0: Status 404 returned error can't find the container with id 7da3a3b4637ec8056f3010f3fd0125f547ec4fefe54dced2214ac5990c0ed5e0 Nov 02 22:58:09.733544 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:09.733498 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4tbpp" event={"ID":"818b5c39-d141-455e-9ff2-ff03b4ecc69d","Type":"ContainerStarted","Data":"7da3a3b4637ec8056f3010f3fd0125f547ec4fefe54dced2214ac5990c0ed5e0"} Nov 02 22:58:09.881643 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:58:09.881610 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 22:58:16.771366 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:16.771328 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4tbpp" event={"ID":"818b5c39-d141-455e-9ff2-ff03b4ecc69d","Type":"ContainerStarted","Data":"bdc8d163678654ad7f44fbdd0b28eff081d5a13205880a10a3121f9180181b23"} Nov 02 22:58:16.771780 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:16.771408 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4tbpp" Nov 02 22:58:16.879372 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:16.879320 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4tbpp" podStartSLOduration=1.6701165900000001 podStartE2EDuration="8.879305919s" totalImagesPullingTime="7.209189329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:58:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:58:16.878236108 +0000 UTC m=+345.846767891" watchObservedRunningTime="2026-11-02 22:58:16.879305919 +0000 UTC m=+345.847837703" Nov 02 22:58:17.668991 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:17.668958 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hc5jd" Nov 02 22:58:22.805489 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:22.805457 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4tbpp" Nov 02 22:58:39.883007 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:58:39.882966 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 22:58:53.251045 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:53.251015 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trmks"] Nov 02 22:58:53.276787 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:53.276761 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trmks" Nov 02 22:58:53.326669 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:53.326634 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 22:58:53.326859 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:53.326847 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 22:58:53.326933 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:53.326848 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 22:58:53.374343 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:53.374315 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-tbzbd\"" Nov 02 22:58:53.376068 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:53.376048 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trmks"] Nov 02 22:58:53.415790 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:53.415754 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f68c4de-4657-4917-a910-72f4b56b39e3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trmks\" (UID: \"6f68c4de-4657-4917-a910-72f4b56b39e3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trmks" Nov 02 22:58:53.415790 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:53.415791 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6f68c4de-4657-4917-a910-72f4b56b39e3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trmks\" (UID: \"6f68c4de-4657-4917-a910-72f4b56b39e3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trmks" Nov 02 22:58:53.416032 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:53.415815 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l226t\" (UniqueName: \"kubernetes.io/projected/6f68c4de-4657-4917-a910-72f4b56b39e3-kube-api-access-l226t\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trmks\" (UID: \"6f68c4de-4657-4917-a910-72f4b56b39e3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trmks" Nov 02 22:58:53.416032 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:53.415918 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6f68c4de-4657-4917-a910-72f4b56b39e3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trmks\" (UID: \"6f68c4de-4657-4917-a910-72f4b56b39e3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trmks" Nov 02 22:58:53.416032 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:53.415973 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6f68c4de-4657-4917-a910-72f4b56b39e3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trmks\" (UID: \"6f68c4de-4657-4917-a910-72f4b56b39e3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trmks" Nov 02 22:58:53.416032 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:53.416020 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f68c4de-4657-4917-a910-72f4b56b39e3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trmks\" (UID: \"6f68c4de-4657-4917-a910-72f4b56b39e3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trmks" Nov 02 22:58:53.416179 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:53.416047 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6f68c4de-4657-4917-a910-72f4b56b39e3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trmks\" (UID: \"6f68c4de-4657-4917-a910-72f4b56b39e3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trmks" Nov 02 22:58:53.516757 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:53.516677 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f68c4de-4657-4917-a910-72f4b56b39e3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trmks\" (UID: \"6f68c4de-4657-4917-a910-72f4b56b39e3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trmks" Nov 02 22:58:53.516757 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:53.516714 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6f68c4de-4657-4917-a910-72f4b56b39e3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trmks\" (UID: \"6f68c4de-4657-4917-a910-72f4b56b39e3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trmks" Nov 02 22:58:53.516757 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:53.516744 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l226t\" (UniqueName: \"kubernetes.io/projected/6f68c4de-4657-4917-a910-72f4b56b39e3-kube-api-access-l226t\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trmks\" (UID: \"6f68c4de-4657-4917-a910-72f4b56b39e3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trmks" Nov 02 22:58:53.517058 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:53.516770 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6f68c4de-4657-4917-a910-72f4b56b39e3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trmks\" (UID: \"6f68c4de-4657-4917-a910-72f4b56b39e3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trmks" Nov 02 22:58:53.517058 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:53.516798 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6f68c4de-4657-4917-a910-72f4b56b39e3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trmks\" (UID: \"6f68c4de-4657-4917-a910-72f4b56b39e3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trmks" Nov 02 22:58:53.517058 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:53.516828 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f68c4de-4657-4917-a910-72f4b56b39e3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trmks\" (UID: \"6f68c4de-4657-4917-a910-72f4b56b39e3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trmks" Nov 02 22:58:53.517058 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:53.516862 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6f68c4de-4657-4917-a910-72f4b56b39e3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trmks\" (UID: \"6f68c4de-4657-4917-a910-72f4b56b39e3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trmks" Nov 02 22:58:53.517714 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:53.517684 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6f68c4de-4657-4917-a910-72f4b56b39e3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trmks\" (UID: \"6f68c4de-4657-4917-a910-72f4b56b39e3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trmks" Nov 02 22:58:53.519197 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:53.519168 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6f68c4de-4657-4917-a910-72f4b56b39e3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trmks\" (UID: \"6f68c4de-4657-4917-a910-72f4b56b39e3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trmks" Nov 02 22:58:53.519319 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:53.519292 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f68c4de-4657-4917-a910-72f4b56b39e3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trmks\" (UID: \"6f68c4de-4657-4917-a910-72f4b56b39e3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trmks" Nov 02 22:58:53.519372 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:53.519339 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6f68c4de-4657-4917-a910-72f4b56b39e3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trmks\" (UID: \"6f68c4de-4657-4917-a910-72f4b56b39e3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trmks" Nov 02 22:58:53.519372 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:53.519350 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6f68c4de-4657-4917-a910-72f4b56b39e3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trmks\" (UID: \"6f68c4de-4657-4917-a910-72f4b56b39e3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trmks" Nov 02 22:58:53.572246 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:53.572206 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l226t\" (UniqueName: \"kubernetes.io/projected/6f68c4de-4657-4917-a910-72f4b56b39e3-kube-api-access-l226t\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trmks\" (UID: \"6f68c4de-4657-4917-a910-72f4b56b39e3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trmks" Nov 02 22:58:53.572381 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:53.572246 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f68c4de-4657-4917-a910-72f4b56b39e3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-trmks\" (UID: \"6f68c4de-4657-4917-a910-72f4b56b39e3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trmks" Nov 02 22:58:53.590231 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:53.590205 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trmks" Nov 02 22:58:54.003598 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:54.003555 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trmks"] Nov 02 22:58:54.038794 ip-10-0-130-211 kubenswrapper[5076]: W1102 22:58:54.038765 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f68c4de_4657_4917_a910_72f4b56b39e3.slice/crio-a73dcf311da10453a5529d01111efa278e600d6ee926e8dacaa4b633bccc7f81 WatchSource:0}: Error finding container a73dcf311da10453a5529d01111efa278e600d6ee926e8dacaa4b633bccc7f81: Status 404 returned error can't find the container with id a73dcf311da10453a5529d01111efa278e600d6ee926e8dacaa4b633bccc7f81 Nov 02 22:58:54.949904 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:54.949858 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trmks" event={"ID":"6f68c4de-4657-4917-a910-72f4b56b39e3","Type":"ContainerStarted","Data":"a73dcf311da10453a5529d01111efa278e600d6ee926e8dacaa4b633bccc7f81"} Nov 02 22:58:57.907797 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:57.907766 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:58:57.908127 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:57.907832 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:58:58.971638 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:58.971598 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trmks" event={"ID":"6f68c4de-4657-4917-a910-72f4b56b39e3","Type":"ContainerStarted","Data":"d9820e52b06e4d4e2c918e67c635420487e6206ccf9d73debb9937c3c479a775"} Nov 02 22:58:58.972138 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:58.971746 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trmks" Nov 02 22:58:58.973392 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:58.973367 5076 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-trmks container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:58:58.973522 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:58.973425 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trmks" podUID="6f68c4de-4657-4917-a910-72f4b56b39e3" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:58:59.183941 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:59.183856 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trmks" podStartSLOduration=2.317838752 podStartE2EDuration="6.183837052s" totalImagesPullingTime="3.8659983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:58:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:58:59.134399426 +0000 UTC m=+388.102931229" watchObservedRunningTime="2026-11-02 22:58:59.183837052 +0000 UTC m=+388.152368837" Nov 02 22:58:59.976999 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:58:59.976970 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-trmks" Nov 02 22:59:09.882272 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:59:09.882232 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 22:59:35.435390 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:59:35.435351 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-lmk64"] Nov 02 22:59:35.436033 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:59:35.435591 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-lmk64" podUID="f9575964-4fe1-4130-b639-ceb989948972" containerName="authorino" containerID="cri-o://c6b1679529f58ad5662333cf9e2ed79f6424770500d1701eab9a533675d55ea8" gracePeriod=30 Nov 02 22:59:35.688721 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:59:35.688666 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-lmk64" Nov 02 22:59:35.800864 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:59:35.800831 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9575964-4fe1-4130-b639-ceb989948972-kube-api-access-r8478\" (UniqueName: \"kubernetes.io/projected/f9575964-4fe1-4130-b639-ceb989948972-kube-api-access-r8478\") pod \"f9575964-4fe1-4130-b639-ceb989948972\" (UID: \"f9575964-4fe1-4130-b639-ceb989948972\") " Nov 02 22:59:35.803136 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:59:35.803108 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9575964-4fe1-4130-b639-ceb989948972-kube-api-access-r8478" pod "f9575964-4fe1-4130-b639-ceb989948972" (UID: "f9575964-4fe1-4130-b639-ceb989948972"). InnerVolumeSpecName "kube-api-access-r8478". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:59:35.901805 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:59:35.901754 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8478\" (UniqueName: \"kubernetes.io/projected/f9575964-4fe1-4130-b639-ceb989948972-kube-api-access-r8478\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:59:36.152803 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:59:36.152715 5076 generic.go:356] "Generic (PLEG): container finished" podID="f9575964-4fe1-4130-b639-ceb989948972" containerID="c6b1679529f58ad5662333cf9e2ed79f6424770500d1701eab9a533675d55ea8" exitCode=0 Nov 02 22:59:36.152803 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:59:36.152757 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-lmk64" Nov 02 22:59:36.153040 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:59:36.152798 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-lmk64" event={"ID":"f9575964-4fe1-4130-b639-ceb989948972","Type":"ContainerDied","Data":"c6b1679529f58ad5662333cf9e2ed79f6424770500d1701eab9a533675d55ea8"} Nov 02 22:59:36.153040 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:59:36.152835 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-lmk64" event={"ID":"f9575964-4fe1-4130-b639-ceb989948972","Type":"ContainerDied","Data":"a7a23e0b161f9a80ea8c9163aaaf413afb0c19284e59b14003ea2a7ecc01d712"} Nov 02 22:59:36.153040 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:59:36.152853 5076 scope.go:122] "RemoveContainer" containerID="c6b1679529f58ad5662333cf9e2ed79f6424770500d1701eab9a533675d55ea8" Nov 02 22:59:36.161566 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:59:36.161550 5076 scope.go:122] "RemoveContainer" containerID="c6b1679529f58ad5662333cf9e2ed79f6424770500d1701eab9a533675d55ea8" Nov 02 22:59:36.161853 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:59:36.161833 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6b1679529f58ad5662333cf9e2ed79f6424770500d1701eab9a533675d55ea8\": container with ID starting with c6b1679529f58ad5662333cf9e2ed79f6424770500d1701eab9a533675d55ea8 not found: ID does not exist" containerID="c6b1679529f58ad5662333cf9e2ed79f6424770500d1701eab9a533675d55ea8" Nov 02 22:59:36.162020 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:59:36.161861 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6b1679529f58ad5662333cf9e2ed79f6424770500d1701eab9a533675d55ea8"} err="failed to get container status \"c6b1679529f58ad5662333cf9e2ed79f6424770500d1701eab9a533675d55ea8\": rpc error: code = NotFound desc = could not find container \"c6b1679529f58ad5662333cf9e2ed79f6424770500d1701eab9a533675d55ea8\": container with ID starting with c6b1679529f58ad5662333cf9e2ed79f6424770500d1701eab9a533675d55ea8 not found: ID does not exist" Nov 02 22:59:36.306991 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:59:36.306961 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-lmk64"] Nov 02 22:59:36.310786 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:59:36.310756 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-lmk64"] Nov 02 22:59:37.146465 ip-10-0-130-211 kubenswrapper[5076]: I1102 22:59:37.146434 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9575964-4fe1-4130-b639-ceb989948972" path="/var/lib/kubelet/pods/f9575964-4fe1-4130-b639-ceb989948972/volumes" Nov 02 22:59:39.882292 ip-10-0-130-211 kubenswrapper[5076]: E1102 22:59:39.882248 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:00:09.882007 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:00:09.881966 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:00:39.881776 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:00:39.881737 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:01:09.882723 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:01:09.882633 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:01:39.885663 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:01:39.885623 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:01:47.448156 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:47.448123 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh"] Nov 02 23:01:47.448626 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:47.448523 5076 cpu_manager.go:414] "containerMap: removing container" podUID="f9575964-4fe1-4130-b639-ceb989948972" containerName="authorino" Nov 02 23:01:47.448626 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:47.448534 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9575964-4fe1-4130-b639-ceb989948972" containerName="authorino" Nov 02 23:01:47.448626 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:47.448595 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9575964-4fe1-4130-b639-ceb989948972" containerName="authorino" Nov 02 23:01:47.450676 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:47.450656 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh" Nov 02 23:01:47.550745 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:47.550712 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 23:01:47.551018 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:47.551000 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 23:01:47.551122 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:47.551062 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 02 23:01:47.551174 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:47.551133 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-cnbg7\"" Nov 02 23:01:47.552539 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:47.552518 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh"] Nov 02 23:01:47.559417 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:47.559390 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00417fb1-0bb2-48cf-85a4-deca8e7f01a5-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh\" (UID: \"00417fb1-0bb2-48cf-85a4-deca8e7f01a5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh" Nov 02 23:01:47.559535 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:47.559429 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs4d2\" (UniqueName: \"kubernetes.io/projected/00417fb1-0bb2-48cf-85a4-deca8e7f01a5-kube-api-access-qs4d2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh\" (UID: \"00417fb1-0bb2-48cf-85a4-deca8e7f01a5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh" Nov 02 23:01:47.559535 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:47.559459 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00417fb1-0bb2-48cf-85a4-deca8e7f01a5-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh\" (UID: \"00417fb1-0bb2-48cf-85a4-deca8e7f01a5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh" Nov 02 23:01:47.559672 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:47.559548 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00417fb1-0bb2-48cf-85a4-deca8e7f01a5-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh\" (UID: \"00417fb1-0bb2-48cf-85a4-deca8e7f01a5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh" Nov 02 23:01:47.559672 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:47.559578 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00417fb1-0bb2-48cf-85a4-deca8e7f01a5-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh\" (UID: \"00417fb1-0bb2-48cf-85a4-deca8e7f01a5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh" Nov 02 23:01:47.559672 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:47.559643 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00417fb1-0bb2-48cf-85a4-deca8e7f01a5-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh\" (UID: \"00417fb1-0bb2-48cf-85a4-deca8e7f01a5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh" Nov 02 23:01:47.660232 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:47.660191 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00417fb1-0bb2-48cf-85a4-deca8e7f01a5-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh\" (UID: \"00417fb1-0bb2-48cf-85a4-deca8e7f01a5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh" Nov 02 23:01:47.660434 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:47.660257 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs4d2\" (UniqueName: \"kubernetes.io/projected/00417fb1-0bb2-48cf-85a4-deca8e7f01a5-kube-api-access-qs4d2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh\" (UID: \"00417fb1-0bb2-48cf-85a4-deca8e7f01a5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh" Nov 02 23:01:47.660434 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:47.660293 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00417fb1-0bb2-48cf-85a4-deca8e7f01a5-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh\" (UID: \"00417fb1-0bb2-48cf-85a4-deca8e7f01a5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh" Nov 02 23:01:47.660434 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:47.660339 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00417fb1-0bb2-48cf-85a4-deca8e7f01a5-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh\" (UID: \"00417fb1-0bb2-48cf-85a4-deca8e7f01a5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh" Nov 02 23:01:47.660434 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:47.660366 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00417fb1-0bb2-48cf-85a4-deca8e7f01a5-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh\" (UID: \"00417fb1-0bb2-48cf-85a4-deca8e7f01a5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh" Nov 02 23:01:47.660434 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:47.660416 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00417fb1-0bb2-48cf-85a4-deca8e7f01a5-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh\" (UID: \"00417fb1-0bb2-48cf-85a4-deca8e7f01a5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh" Nov 02 23:01:47.660976 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:47.660941 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00417fb1-0bb2-48cf-85a4-deca8e7f01a5-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh\" (UID: \"00417fb1-0bb2-48cf-85a4-deca8e7f01a5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh" Nov 02 23:01:47.661328 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:47.661305 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00417fb1-0bb2-48cf-85a4-deca8e7f01a5-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh\" (UID: \"00417fb1-0bb2-48cf-85a4-deca8e7f01a5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh" Nov 02 23:01:47.661496 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:47.661478 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00417fb1-0bb2-48cf-85a4-deca8e7f01a5-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh\" (UID: \"00417fb1-0bb2-48cf-85a4-deca8e7f01a5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh" Nov 02 23:01:47.662965 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:47.662938 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00417fb1-0bb2-48cf-85a4-deca8e7f01a5-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh\" (UID: \"00417fb1-0bb2-48cf-85a4-deca8e7f01a5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh" Nov 02 23:01:47.663739 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:47.663721 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00417fb1-0bb2-48cf-85a4-deca8e7f01a5-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh\" (UID: \"00417fb1-0bb2-48cf-85a4-deca8e7f01a5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh" Nov 02 23:01:47.714989 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:47.714913 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs4d2\" (UniqueName: \"kubernetes.io/projected/00417fb1-0bb2-48cf-85a4-deca8e7f01a5-kube-api-access-qs4d2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh\" (UID: \"00417fb1-0bb2-48cf-85a4-deca8e7f01a5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh" Nov 02 23:01:47.762653 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:47.762623 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh" Nov 02 23:01:48.139371 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:48.139342 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh"] Nov 02 23:01:48.141661 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:01:48.141637 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00417fb1_0bb2_48cf_85a4_deca8e7f01a5.slice/crio-3d1b34abedb557dafc9ef47573c7e50020ff4e28792dfd8b029525e64c2e0404 WatchSource:0}: Error finding container 3d1b34abedb557dafc9ef47573c7e50020ff4e28792dfd8b029525e64c2e0404: Status 404 returned error can't find the container with id 3d1b34abedb557dafc9ef47573c7e50020ff4e28792dfd8b029525e64c2e0404 Nov 02 23:01:48.775586 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:48.775542 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh" event={"ID":"00417fb1-0bb2-48cf-85a4-deca8e7f01a5","Type":"ContainerStarted","Data":"3d1b34abedb557dafc9ef47573c7e50020ff4e28792dfd8b029525e64c2e0404"} Nov 02 23:01:49.782083 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:49.782044 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh" event={"ID":"00417fb1-0bb2-48cf-85a4-deca8e7f01a5","Type":"ContainerStarted","Data":"488dd29af3c8ea1335c299cb5a7fef39b6b241c65f0709e7cedd4e1fd02da595"} Nov 02 23:01:49.892230 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:49.892135 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh" podStartSLOduration=1.448903871 podStartE2EDuration="2.892121591s" totalImagesPullingTime="1.44321772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:01:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:01:49.890713526 +0000 UTC m=+558.859245331" watchObservedRunningTime="2026-11-02 23:01:49.892121591 +0000 UTC m=+558.860653374" Nov 02 23:01:50.763349 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:50.763315 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh" Nov 02 23:01:54.476997 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:54.476962 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw"] Nov 02 23:01:54.480720 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:54.480700 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw" Nov 02 23:01:54.521184 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:54.521154 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a0c3861-1c5a-4155-ae4b-785129cd0f1e-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw\" (UID: \"5a0c3861-1c5a-4155-ae4b-785129cd0f1e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw" Nov 02 23:01:54.521334 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:54.521198 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a0c3861-1c5a-4155-ae4b-785129cd0f1e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw\" (UID: \"5a0c3861-1c5a-4155-ae4b-785129cd0f1e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw" Nov 02 23:01:54.521334 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:54.521225 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a0c3861-1c5a-4155-ae4b-785129cd0f1e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw\" (UID: \"5a0c3861-1c5a-4155-ae4b-785129cd0f1e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw" Nov 02 23:01:54.521427 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:54.521358 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a0c3861-1c5a-4155-ae4b-785129cd0f1e-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw\" (UID: \"5a0c3861-1c5a-4155-ae4b-785129cd0f1e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw" Nov 02 23:01:54.521427 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:54.521393 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pshc4\" (UniqueName: \"kubernetes.io/projected/5a0c3861-1c5a-4155-ae4b-785129cd0f1e-kube-api-access-pshc4\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw\" (UID: \"5a0c3861-1c5a-4155-ae4b-785129cd0f1e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw" Nov 02 23:01:54.521427 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:54.521414 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a0c3861-1c5a-4155-ae4b-785129cd0f1e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw\" (UID: \"5a0c3861-1c5a-4155-ae4b-785129cd0f1e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw" Nov 02 23:01:54.534506 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:54.534480 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw"] Nov 02 23:01:54.579111 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:54.579080 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 02 23:01:54.622316 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:54.622282 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a0c3861-1c5a-4155-ae4b-785129cd0f1e-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw\" (UID: \"5a0c3861-1c5a-4155-ae4b-785129cd0f1e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw" Nov 02 23:01:54.622608 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:54.622572 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a0c3861-1c5a-4155-ae4b-785129cd0f1e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw\" (UID: \"5a0c3861-1c5a-4155-ae4b-785129cd0f1e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw" Nov 02 23:01:54.622755 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:54.622635 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a0c3861-1c5a-4155-ae4b-785129cd0f1e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw\" (UID: \"5a0c3861-1c5a-4155-ae4b-785129cd0f1e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw" Nov 02 23:01:54.622755 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:54.622682 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a0c3861-1c5a-4155-ae4b-785129cd0f1e-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw\" (UID: \"5a0c3861-1c5a-4155-ae4b-785129cd0f1e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw" Nov 02 23:01:54.622755 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:54.622703 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pshc4\" (UniqueName: \"kubernetes.io/projected/5a0c3861-1c5a-4155-ae4b-785129cd0f1e-kube-api-access-pshc4\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw\" (UID: \"5a0c3861-1c5a-4155-ae4b-785129cd0f1e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw" Nov 02 23:01:54.622755 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:54.622728 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a0c3861-1c5a-4155-ae4b-785129cd0f1e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw\" (UID: \"5a0c3861-1c5a-4155-ae4b-785129cd0f1e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw" Nov 02 23:01:54.624105 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:54.624064 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a0c3861-1c5a-4155-ae4b-785129cd0f1e-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw\" (UID: \"5a0c3861-1c5a-4155-ae4b-785129cd0f1e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw" Nov 02 23:01:54.624105 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:54.624093 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a0c3861-1c5a-4155-ae4b-785129cd0f1e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw\" (UID: \"5a0c3861-1c5a-4155-ae4b-785129cd0f1e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw" Nov 02 23:01:54.624301 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:54.624138 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a0c3861-1c5a-4155-ae4b-785129cd0f1e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw\" (UID: \"5a0c3861-1c5a-4155-ae4b-785129cd0f1e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw" Nov 02 23:01:54.625959 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:54.625927 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a0c3861-1c5a-4155-ae4b-785129cd0f1e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw\" (UID: \"5a0c3861-1c5a-4155-ae4b-785129cd0f1e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw" Nov 02 23:01:54.627788 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:54.627764 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a0c3861-1c5a-4155-ae4b-785129cd0f1e-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw\" (UID: \"5a0c3861-1c5a-4155-ae4b-785129cd0f1e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw" Nov 02 23:01:54.680980 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:54.680950 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pshc4\" (UniqueName: \"kubernetes.io/projected/5a0c3861-1c5a-4155-ae4b-785129cd0f1e-kube-api-access-pshc4\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw\" (UID: \"5a0c3861-1c5a-4155-ae4b-785129cd0f1e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw" Nov 02 23:01:54.793371 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:54.793275 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw" Nov 02 23:01:55.162950 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:55.162921 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw"] Nov 02 23:01:55.164676 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:01:55.164643 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a0c3861_1c5a_4155_ae4b_785129cd0f1e.slice/crio-c3916fc49a5abbe9f232789838fc8c5c6307f912ab5d54dc951847d0c420c128 WatchSource:0}: Error finding container c3916fc49a5abbe9f232789838fc8c5c6307f912ab5d54dc951847d0c420c128: Status 404 returned error can't find the container with id c3916fc49a5abbe9f232789838fc8c5c6307f912ab5d54dc951847d0c420c128 Nov 02 23:01:55.811254 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:55.811223 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw" event={"ID":"5a0c3861-1c5a-4155-ae4b-785129cd0f1e","Type":"ContainerStarted","Data":"c3916fc49a5abbe9f232789838fc8c5c6307f912ab5d54dc951847d0c420c128"} Nov 02 23:01:56.817989 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:56.817948 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw" event={"ID":"5a0c3861-1c5a-4155-ae4b-785129cd0f1e","Type":"ContainerStarted","Data":"536cc0cde9d92a738cb60c5825d88cc7582bf1f061ab4dae0959f0b66e308206"} Nov 02 23:01:56.930710 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:56.930653 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw" podStartSLOduration=2.322973075 podStartE2EDuration="2.930640289s" totalImagesPullingTime="607.667214ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:01:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:01:56.928473268 +0000 UTC m=+565.897005055" watchObservedRunningTime="2026-11-02 23:01:56.930640289 +0000 UTC m=+565.899172072" Nov 02 23:01:57.357091 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:57.357059 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8"] Nov 02 23:01:57.360639 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:57.360619 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" Nov 02 23:01:57.410496 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:57.410462 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-tbbgz\"" Nov 02 23:01:57.451274 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:57.451241 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v7jp8\" (UID: \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" Nov 02 23:01:57.451274 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:57.451285 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v7jp8\" (UID: \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" Nov 02 23:01:57.451572 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:57.451313 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v7jp8\" (UID: \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" Nov 02 23:01:57.451760 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:57.451731 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v7jp8\" (UID: \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" Nov 02 23:01:57.451995 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:57.451963 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v7jp8\" (UID: \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" Nov 02 23:01:57.452100 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:57.451996 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v7jp8\" (UID: \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" Nov 02 23:01:57.452100 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:57.452017 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg9g2\" (UniqueName: \"kubernetes.io/projected/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-kube-api-access-dg9g2\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v7jp8\" (UID: \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" Nov 02 23:01:57.452197 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:57.452171 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v7jp8\" (UID: \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" Nov 02 23:01:57.452244 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:57.452213 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v7jp8\" (UID: \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" Nov 02 23:01:57.460684 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:57.460655 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8"] Nov 02 23:01:57.553082 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:57.553047 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v7jp8\" (UID: \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" Nov 02 23:01:57.553272 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:57.553102 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v7jp8\" (UID: \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" Nov 02 23:01:57.553272 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:57.553120 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v7jp8\" (UID: \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" Nov 02 23:01:57.553272 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:57.553142 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg9g2\" (UniqueName: \"kubernetes.io/projected/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-kube-api-access-dg9g2\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v7jp8\" (UID: \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" Nov 02 23:01:57.553272 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:57.553208 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v7jp8\" (UID: \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" Nov 02 23:01:57.553272 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:57.553233 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v7jp8\" (UID: \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" Nov 02 23:01:57.553272 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:57.553258 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v7jp8\" (UID: \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" Nov 02 23:01:57.553584 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:57.553283 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v7jp8\" (UID: \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" Nov 02 23:01:57.553584 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:57.553417 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v7jp8\" (UID: \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" Nov 02 23:01:57.553584 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:57.553556 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v7jp8\" (UID: \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" Nov 02 23:01:57.553748 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:57.553658 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v7jp8\" (UID: \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" Nov 02 23:01:57.553748 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:57.553720 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v7jp8\" (UID: \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" Nov 02 23:01:57.553976 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:57.553946 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v7jp8\" (UID: \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" Nov 02 23:01:57.554105 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:57.553979 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v7jp8\" (UID: \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" Nov 02 23:01:57.555605 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:57.555584 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v7jp8\" (UID: \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" Nov 02 23:01:57.555867 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:57.555851 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v7jp8\" (UID: \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" Nov 02 23:01:57.607615 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:57.607544 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v7jp8\" (UID: \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" Nov 02 23:01:57.607734 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:57.607640 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg9g2\" (UniqueName: \"kubernetes.io/projected/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-kube-api-access-dg9g2\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v7jp8\" (UID: \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" Nov 02 23:01:57.673516 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:57.673481 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" Nov 02 23:01:57.763333 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:57.763298 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh" Nov 02 23:01:57.776164 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:57.776138 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh" Nov 02 23:01:57.794066 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:57.794038 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw" Nov 02 23:01:57.834806 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:57.834776 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh" Nov 02 23:01:58.087618 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:58.087593 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8"] Nov 02 23:01:58.089472 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:01:58.089443 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad898fd0_2ba1_4e99_ae5a_14dd3afb3d8d.slice/crio-80c2beead924f4047ba6c1ec29fd0aa93946be3999c044f7d210ceee15d6f348 WatchSource:0}: Error finding container 80c2beead924f4047ba6c1ec29fd0aa93946be3999c044f7d210ceee15d6f348: Status 404 returned error can't find the container with id 80c2beead924f4047ba6c1ec29fd0aa93946be3999c044f7d210ceee15d6f348 Nov 02 23:01:58.091725 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:58.091697 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:01:58.091797 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:58.091757 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:01:58.091797 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:58.091787 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:01:58.844251 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:58.844199 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" event={"ID":"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d","Type":"ContainerStarted","Data":"271338a4e786084b47c46ed1176d93ad1ca308b53f49763c1bb02adac63ab9d7"} Nov 02 23:01:58.844772 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:58.844273 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" event={"ID":"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d","Type":"ContainerStarted","Data":"80c2beead924f4047ba6c1ec29fd0aa93946be3999c044f7d210ceee15d6f348"} Nov 02 23:01:58.956269 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:58.956215 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" podStartSLOduration=1.956200473 podStartE2EDuration="1.956200473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:01:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:01:58.954734455 +0000 UTC m=+567.923266238" watchObservedRunningTime="2026-11-02 23:01:58.956200473 +0000 UTC m=+567.924732253" Nov 02 23:01:59.674258 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:59.674209 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" Nov 02 23:01:59.675859 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:59.675835 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Nov 02 23:01:59.675948 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:01:59.675914 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" podUID="ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Nov 02 23:02:00.674125 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:00.674081 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Nov 02 23:02:00.674514 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:00.674145 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" podUID="ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Nov 02 23:02:01.674673 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:01.674634 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Nov 02 23:02:01.675093 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:01.674692 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" podUID="ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Nov 02 23:02:02.674939 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:02.674871 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Nov 02 23:02:02.675436 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:02.674962 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" podUID="ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Nov 02 23:02:03.674635 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:03.674597 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Nov 02 23:02:03.674923 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:03.674650 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" podUID="ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Nov 02 23:02:04.674914 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:04.674854 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Nov 02 23:02:04.675325 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:04.674943 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" podUID="ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Nov 02 23:02:04.793743 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:04.793703 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw" Nov 02 23:02:04.812816 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:04.812787 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw" Nov 02 23:02:04.883448 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:04.883418 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw" Nov 02 23:02:06.008941 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:06.008884 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" Nov 02 23:02:06.009372 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:06.009277 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" Nov 02 23:02:06.010194 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:06.010175 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" Nov 02 23:02:09.883712 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:02:09.883670 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:02:12.489781 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:12.489680 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z"] Nov 02 23:02:12.493849 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:12.493825 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" Nov 02 23:02:12.582304 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:12.582267 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z"] Nov 02 23:02:12.621244 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:12.621215 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b820762-bd67-4d19-a050-224c6225fe67-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sc89z\" (UID: \"9b820762-bd67-4d19-a050-224c6225fe67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" Nov 02 23:02:12.621414 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:12.621251 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b820762-bd67-4d19-a050-224c6225fe67-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sc89z\" (UID: \"9b820762-bd67-4d19-a050-224c6225fe67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" Nov 02 23:02:12.621414 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:12.621277 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b820762-bd67-4d19-a050-224c6225fe67-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sc89z\" (UID: \"9b820762-bd67-4d19-a050-224c6225fe67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" Nov 02 23:02:12.621414 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:12.621389 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b820762-bd67-4d19-a050-224c6225fe67-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sc89z\" (UID: \"9b820762-bd67-4d19-a050-224c6225fe67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" Nov 02 23:02:12.621515 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:12.621471 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b820762-bd67-4d19-a050-224c6225fe67-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sc89z\" (UID: \"9b820762-bd67-4d19-a050-224c6225fe67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" Nov 02 23:02:12.621556 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:12.621543 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8mr7\" (UniqueName: \"kubernetes.io/projected/9b820762-bd67-4d19-a050-224c6225fe67-kube-api-access-s8mr7\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sc89z\" (UID: \"9b820762-bd67-4d19-a050-224c6225fe67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" Nov 02 23:02:12.621589 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:12.621568 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b820762-bd67-4d19-a050-224c6225fe67-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sc89z\" (UID: \"9b820762-bd67-4d19-a050-224c6225fe67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" Nov 02 23:02:12.621622 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:12.621588 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b820762-bd67-4d19-a050-224c6225fe67-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sc89z\" (UID: \"9b820762-bd67-4d19-a050-224c6225fe67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" Nov 02 23:02:12.621622 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:12.621614 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b820762-bd67-4d19-a050-224c6225fe67-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sc89z\" (UID: \"9b820762-bd67-4d19-a050-224c6225fe67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" Nov 02 23:02:12.722534 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:12.722495 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b820762-bd67-4d19-a050-224c6225fe67-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sc89z\" (UID: \"9b820762-bd67-4d19-a050-224c6225fe67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" Nov 02 23:02:12.722534 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:12.722540 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8mr7\" (UniqueName: \"kubernetes.io/projected/9b820762-bd67-4d19-a050-224c6225fe67-kube-api-access-s8mr7\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sc89z\" (UID: \"9b820762-bd67-4d19-a050-224c6225fe67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" Nov 02 23:02:12.722780 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:12.722577 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b820762-bd67-4d19-a050-224c6225fe67-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sc89z\" (UID: \"9b820762-bd67-4d19-a050-224c6225fe67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" Nov 02 23:02:12.722780 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:12.722597 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b820762-bd67-4d19-a050-224c6225fe67-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sc89z\" (UID: \"9b820762-bd67-4d19-a050-224c6225fe67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" Nov 02 23:02:12.722780 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:12.722625 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b820762-bd67-4d19-a050-224c6225fe67-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sc89z\" (UID: \"9b820762-bd67-4d19-a050-224c6225fe67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" Nov 02 23:02:12.722780 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:12.722673 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b820762-bd67-4d19-a050-224c6225fe67-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sc89z\" (UID: \"9b820762-bd67-4d19-a050-224c6225fe67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" Nov 02 23:02:12.722780 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:12.722703 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b820762-bd67-4d19-a050-224c6225fe67-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sc89z\" (UID: \"9b820762-bd67-4d19-a050-224c6225fe67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" Nov 02 23:02:12.722780 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:12.722747 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b820762-bd67-4d19-a050-224c6225fe67-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sc89z\" (UID: \"9b820762-bd67-4d19-a050-224c6225fe67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" Nov 02 23:02:12.723106 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:12.722979 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b820762-bd67-4d19-a050-224c6225fe67-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sc89z\" (UID: \"9b820762-bd67-4d19-a050-224c6225fe67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" Nov 02 23:02:12.723106 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:12.723015 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b820762-bd67-4d19-a050-224c6225fe67-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sc89z\" (UID: \"9b820762-bd67-4d19-a050-224c6225fe67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" Nov 02 23:02:12.723106 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:12.723030 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b820762-bd67-4d19-a050-224c6225fe67-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sc89z\" (UID: \"9b820762-bd67-4d19-a050-224c6225fe67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" Nov 02 23:02:12.723106 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:12.723075 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b820762-bd67-4d19-a050-224c6225fe67-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sc89z\" (UID: \"9b820762-bd67-4d19-a050-224c6225fe67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" Nov 02 23:02:12.723250 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:12.723169 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b820762-bd67-4d19-a050-224c6225fe67-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sc89z\" (UID: \"9b820762-bd67-4d19-a050-224c6225fe67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" Nov 02 23:02:12.723640 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:12.723613 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b820762-bd67-4d19-a050-224c6225fe67-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sc89z\" (UID: \"9b820762-bd67-4d19-a050-224c6225fe67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" Nov 02 23:02:12.725040 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:12.725016 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b820762-bd67-4d19-a050-224c6225fe67-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sc89z\" (UID: \"9b820762-bd67-4d19-a050-224c6225fe67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" Nov 02 23:02:12.725197 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:12.725180 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b820762-bd67-4d19-a050-224c6225fe67-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sc89z\" (UID: \"9b820762-bd67-4d19-a050-224c6225fe67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" Nov 02 23:02:12.776860 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:12.776781 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b820762-bd67-4d19-a050-224c6225fe67-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sc89z\" (UID: \"9b820762-bd67-4d19-a050-224c6225fe67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" Nov 02 23:02:12.777031 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:12.777006 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8mr7\" (UniqueName: \"kubernetes.io/projected/9b820762-bd67-4d19-a050-224c6225fe67-kube-api-access-s8mr7\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sc89z\" (UID: \"9b820762-bd67-4d19-a050-224c6225fe67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" Nov 02 23:02:12.806437 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:12.806409 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" Nov 02 23:02:13.218677 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:13.218647 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z"] Nov 02 23:02:13.220387 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:02:13.220360 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b820762_bd67_4d19_a050_224c6225fe67.slice/crio-760a2669742b64f85a026c9e62ecc018e0ac5e72ec49524684067b2fbdc507d5 WatchSource:0}: Error finding container 760a2669742b64f85a026c9e62ecc018e0ac5e72ec49524684067b2fbdc507d5: Status 404 returned error can't find the container with id 760a2669742b64f85a026c9e62ecc018e0ac5e72ec49524684067b2fbdc507d5 Nov 02 23:02:13.222553 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:13.222526 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:02:13.222626 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:13.222598 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:02:13.222666 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:13.222627 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:02:13.939022 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:13.938984 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" event={"ID":"9b820762-bd67-4d19-a050-224c6225fe67","Type":"ContainerStarted","Data":"7723c5dbb3f60e1d039437c32bc35fdf00bc8c48b4868e4db88417196782c66a"} Nov 02 23:02:13.939453 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:13.939043 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" event={"ID":"9b820762-bd67-4d19-a050-224c6225fe67","Type":"ContainerStarted","Data":"760a2669742b64f85a026c9e62ecc018e0ac5e72ec49524684067b2fbdc507d5"} Nov 02 23:02:14.144087 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:14.144032 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" podStartSLOduration=2.144018579 podStartE2EDuration="2.144018579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:02:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:02:14.096797467 +0000 UTC m=+583.065329252" watchObservedRunningTime="2026-11-02 23:02:14.144018579 +0000 UTC m=+583.112550363" Nov 02 23:02:14.807454 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:14.807416 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" Nov 02 23:02:14.808904 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:14.808869 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-sc89z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 02 23:02:14.809003 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:14.808961 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" podUID="9b820762-bd67-4d19-a050-224c6225fe67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 02 23:02:15.807218 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:15.807181 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-sc89z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 02 23:02:15.807618 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:15.807235 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" podUID="9b820762-bd67-4d19-a050-224c6225fe67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 02 23:02:16.807344 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:16.807305 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-sc89z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 02 23:02:16.807765 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:16.807362 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" podUID="9b820762-bd67-4d19-a050-224c6225fe67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 02 23:02:17.807804 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:17.807773 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-sc89z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 02 23:02:17.808192 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:17.807841 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" podUID="9b820762-bd67-4d19-a050-224c6225fe67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 02 23:02:18.807026 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:18.806990 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-sc89z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 02 23:02:18.807180 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:18.807044 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" podUID="9b820762-bd67-4d19-a050-224c6225fe67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 02 23:02:19.807462 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:19.807419 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-sc89z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 02 23:02:19.807986 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:19.807492 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" podUID="9b820762-bd67-4d19-a050-224c6225fe67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 02 23:02:21.529022 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:21.528984 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" Nov 02 23:02:21.529508 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:21.529347 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" Nov 02 23:02:21.530302 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:21.530280 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" Nov 02 23:02:31.221849 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:31.221815 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 02 23:02:31.226251 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:02:31.226228 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 02 23:02:39.883834 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:02:39.883789 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:03:03.504450 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:03.504414 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8a68c823-cfbd576f9-ffck9"] Nov 02 23:03:03.507177 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:03.507156 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8a68c823-cfbd576f9-ffck9" Nov 02 23:03:03.529014 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:03.528990 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/061b5c4a-b850-433e-ba62-3ae8de255048-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-8a68c823-cfbd576f9-ffck9\" (UID: \"061b5c4a-b850-433e-ba62-3ae8de255048\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8a68c823-cfbd576f9-ffck9" Nov 02 23:03:03.529134 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:03.529044 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85jcz\" (UniqueName: \"kubernetes.io/projected/061b5c4a-b850-433e-ba62-3ae8de255048-kube-api-access-85jcz\") pod \"payload-pre-processing-e2e-migrate-8a68c823-cfbd576f9-ffck9\" (UID: \"061b5c4a-b850-433e-ba62-3ae8de255048\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8a68c823-cfbd576f9-ffck9" Nov 02 23:03:03.557120 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:03.557092 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-8a68c823\"" Nov 02 23:03:03.622135 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:03.622102 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-8a68c823-dockercfg-5wlqn\"" Nov 02 23:03:03.627977 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:03.627943 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8a68c823-cfbd576f9-ffck9"] Nov 02 23:03:03.633400 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:03.630545 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/061b5c4a-b850-433e-ba62-3ae8de255048-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-8a68c823-cfbd576f9-ffck9\" (UID: \"061b5c4a-b850-433e-ba62-3ae8de255048\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8a68c823-cfbd576f9-ffck9" Nov 02 23:03:03.635090 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:03.635061 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85jcz\" (UniqueName: \"kubernetes.io/projected/061b5c4a-b850-433e-ba62-3ae8de255048-kube-api-access-85jcz\") pod \"payload-pre-processing-e2e-migrate-8a68c823-cfbd576f9-ffck9\" (UID: \"061b5c4a-b850-433e-ba62-3ae8de255048\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8a68c823-cfbd576f9-ffck9" Nov 02 23:03:03.635486 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:03.635459 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/061b5c4a-b850-433e-ba62-3ae8de255048-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-8a68c823-cfbd576f9-ffck9\" (UID: \"061b5c4a-b850-433e-ba62-3ae8de255048\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8a68c823-cfbd576f9-ffck9" Nov 02 23:03:03.671311 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:03.671278 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8a68c823-d654c98dc-7v4lm"] Nov 02 23:03:03.673956 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:03.673936 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8a68c823-d654c98dc-7v4lm" Nov 02 23:03:03.716003 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:03.715959 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8a68c823-d654c98dc-7v4lm"] Nov 02 23:03:03.726434 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:03.726396 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85jcz\" (UniqueName: \"kubernetes.io/projected/061b5c4a-b850-433e-ba62-3ae8de255048-kube-api-access-85jcz\") pod \"payload-pre-processing-e2e-migrate-8a68c823-cfbd576f9-ffck9\" (UID: \"061b5c4a-b850-433e-ba62-3ae8de255048\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8a68c823-cfbd576f9-ffck9" Nov 02 23:03:03.736256 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:03.736228 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvc5r\" (UniqueName: \"kubernetes.io/projected/dfebc122-0b39-4c69-aa55-8d4aafa72905-kube-api-access-tvc5r\") pod \"payload-processing-e2e-migrate-8a68c823-d654c98dc-7v4lm\" (UID: \"dfebc122-0b39-4c69-aa55-8d4aafa72905\") " pod="openshift-ingress/payload-processing-e2e-migrate-8a68c823-d654c98dc-7v4lm" Nov 02 23:03:03.736393 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:03.736275 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfebc122-0b39-4c69-aa55-8d4aafa72905-plugins-config-volume\") pod \"payload-processing-e2e-migrate-8a68c823-d654c98dc-7v4lm\" (UID: \"dfebc122-0b39-4c69-aa55-8d4aafa72905\") " pod="openshift-ingress/payload-processing-e2e-migrate-8a68c823-d654c98dc-7v4lm" Nov 02 23:03:03.819677 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:03.819599 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8a68c823-cfbd576f9-ffck9" Nov 02 23:03:03.836864 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:03.836837 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvc5r\" (UniqueName: \"kubernetes.io/projected/dfebc122-0b39-4c69-aa55-8d4aafa72905-kube-api-access-tvc5r\") pod \"payload-processing-e2e-migrate-8a68c823-d654c98dc-7v4lm\" (UID: \"dfebc122-0b39-4c69-aa55-8d4aafa72905\") " pod="openshift-ingress/payload-processing-e2e-migrate-8a68c823-d654c98dc-7v4lm" Nov 02 23:03:03.837037 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:03.836902 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfebc122-0b39-4c69-aa55-8d4aafa72905-plugins-config-volume\") pod \"payload-processing-e2e-migrate-8a68c823-d654c98dc-7v4lm\" (UID: \"dfebc122-0b39-4c69-aa55-8d4aafa72905\") " pod="openshift-ingress/payload-processing-e2e-migrate-8a68c823-d654c98dc-7v4lm" Nov 02 23:03:03.837677 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:03.837651 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfebc122-0b39-4c69-aa55-8d4aafa72905-plugins-config-volume\") pod \"payload-processing-e2e-migrate-8a68c823-d654c98dc-7v4lm\" (UID: \"dfebc122-0b39-4c69-aa55-8d4aafa72905\") " pod="openshift-ingress/payload-processing-e2e-migrate-8a68c823-d654c98dc-7v4lm" Nov 02 23:03:03.892461 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:03.892427 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvc5r\" (UniqueName: \"kubernetes.io/projected/dfebc122-0b39-4c69-aa55-8d4aafa72905-kube-api-access-tvc5r\") pod \"payload-processing-e2e-migrate-8a68c823-d654c98dc-7v4lm\" (UID: \"dfebc122-0b39-4c69-aa55-8d4aafa72905\") " pod="openshift-ingress/payload-processing-e2e-migrate-8a68c823-d654c98dc-7v4lm" Nov 02 23:03:03.987371 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:03.987335 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8a68c823-d654c98dc-7v4lm" Nov 02 23:03:04.199066 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:04.199042 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8a68c823-cfbd576f9-ffck9"] Nov 02 23:03:04.201277 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:03:04.201249 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod061b5c4a_b850_433e_ba62_3ae8de255048.slice/crio-9afe818b5adaa2e443a642dd023b2c96dced2b00ee4da2567a63717167257ed9 WatchSource:0}: Error finding container 9afe818b5adaa2e443a642dd023b2c96dced2b00ee4da2567a63717167257ed9: Status 404 returned error can't find the container with id 9afe818b5adaa2e443a642dd023b2c96dced2b00ee4da2567a63717167257ed9 Nov 02 23:03:04.203209 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:04.203193 5076 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 23:03:04.355796 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:04.355771 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8a68c823-d654c98dc-7v4lm"] Nov 02 23:03:04.357673 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:03:04.357645 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfebc122_0b39_4c69_aa55_8d4aafa72905.slice/crio-bdc2e60f15cad5dcfd1dd1c4a9d6772499f7380059dfc0a302ab507dd81382ae WatchSource:0}: Error finding container bdc2e60f15cad5dcfd1dd1c4a9d6772499f7380059dfc0a302ab507dd81382ae: Status 404 returned error can't find the container with id bdc2e60f15cad5dcfd1dd1c4a9d6772499f7380059dfc0a302ab507dd81382ae Nov 02 23:03:05.187199 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:05.187160 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8a68c823-cfbd576f9-ffck9" event={"ID":"061b5c4a-b850-433e-ba62-3ae8de255048","Type":"ContainerStarted","Data":"9afe818b5adaa2e443a642dd023b2c96dced2b00ee4da2567a63717167257ed9"} Nov 02 23:03:05.188295 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:05.188269 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8a68c823-d654c98dc-7v4lm" event={"ID":"dfebc122-0b39-4c69-aa55-8d4aafa72905","Type":"ContainerStarted","Data":"bdc2e60f15cad5dcfd1dd1c4a9d6772499f7380059dfc0a302ab507dd81382ae"} Nov 02 23:03:05.567326 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:05.567236 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2"] Nov 02 23:03:05.570388 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:05.570362 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2" Nov 02 23:03:05.649834 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:05.649799 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d999e2e2-910d-4c23-b784-31b2836a9c20-metrics-certs\") pod \"maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2\" (UID: \"d999e2e2-910d-4c23-b784-31b2836a9c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2" Nov 02 23:03:05.650025 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:05.649859 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbkqq\" (UniqueName: \"kubernetes.io/projected/d999e2e2-910d-4c23-b784-31b2836a9c20-kube-api-access-vbkqq\") pod \"maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2\" (UID: \"d999e2e2-910d-4c23-b784-31b2836a9c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2" Nov 02 23:03:05.650025 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:05.650001 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d999e2e2-910d-4c23-b784-31b2836a9c20-maas-api-tls\") pod \"maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2\" (UID: \"d999e2e2-910d-4c23-b784-31b2836a9c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2" Nov 02 23:03:05.650207 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:05.650048 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d999e2e2-910d-4c23-b784-31b2836a9c20-openshift-service-ca\") pod \"maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2\" (UID: \"d999e2e2-910d-4c23-b784-31b2836a9c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2" Nov 02 23:03:05.670274 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:05.670243 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 23:03:05.670944 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:05.670924 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-b7gfc\"" Nov 02 23:03:05.671059 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:05.670985 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-8a68c823-serving-cert\"" Nov 02 23:03:05.673318 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:05.673291 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2"] Nov 02 23:03:05.750709 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:05.750666 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d999e2e2-910d-4c23-b784-31b2836a9c20-metrics-certs\") pod \"maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2\" (UID: \"d999e2e2-910d-4c23-b784-31b2836a9c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2" Nov 02 23:03:05.750917 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:05.750767 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbkqq\" (UniqueName: \"kubernetes.io/projected/d999e2e2-910d-4c23-b784-31b2836a9c20-kube-api-access-vbkqq\") pod \"maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2\" (UID: \"d999e2e2-910d-4c23-b784-31b2836a9c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2" Nov 02 23:03:05.750917 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:05.750806 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d999e2e2-910d-4c23-b784-31b2836a9c20-maas-api-tls\") pod \"maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2\" (UID: \"d999e2e2-910d-4c23-b784-31b2836a9c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2" Nov 02 23:03:05.750917 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:05.750842 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d999e2e2-910d-4c23-b784-31b2836a9c20-openshift-service-ca\") pod \"maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2\" (UID: \"d999e2e2-910d-4c23-b784-31b2836a9c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2" Nov 02 23:03:05.751380 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:03:05.751357 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-8a68c823-serving-cert: secret "maas-api-e2e-migrate-8a68c823-serving-cert" not found Nov 02 23:03:05.751506 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:03:05.751426 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d999e2e2-910d-4c23-b784-31b2836a9c20-maas-api-tls podName:d999e2e2-910d-4c23-b784-31b2836a9c20 nodeName:}" failed. No retries permitted until 2026-11-02 23:03:06.251407333 +0000 UTC m=+635.219939108 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d999e2e2-910d-4c23-b784-31b2836a9c20-maas-api-tls") pod "maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2" (UID: "d999e2e2-910d-4c23-b784-31b2836a9c20") : secret "maas-api-e2e-migrate-8a68c823-serving-cert" not found Nov 02 23:03:05.752284 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:05.752264 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d999e2e2-910d-4c23-b784-31b2836a9c20-openshift-service-ca\") pod \"maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2\" (UID: \"d999e2e2-910d-4c23-b784-31b2836a9c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2" Nov 02 23:03:05.761699 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:05.761677 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d999e2e2-910d-4c23-b784-31b2836a9c20-metrics-certs\") pod \"maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2\" (UID: \"d999e2e2-910d-4c23-b784-31b2836a9c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2" Nov 02 23:03:05.837957 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:05.830558 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbkqq\" (UniqueName: \"kubernetes.io/projected/d999e2e2-910d-4c23-b784-31b2836a9c20-kube-api-access-vbkqq\") pod \"maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2\" (UID: \"d999e2e2-910d-4c23-b784-31b2836a9c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2" Nov 02 23:03:06.255557 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:06.255518 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d999e2e2-910d-4c23-b784-31b2836a9c20-maas-api-tls\") pod \"maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2\" (UID: \"d999e2e2-910d-4c23-b784-31b2836a9c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2" Nov 02 23:03:06.270610 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:06.270575 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d999e2e2-910d-4c23-b784-31b2836a9c20-maas-api-tls\") pod \"maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2\" (UID: \"d999e2e2-910d-4c23-b784-31b2836a9c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2" Nov 02 23:03:06.485579 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:06.485537 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2" Nov 02 23:03:06.869445 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:06.869374 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2"] Nov 02 23:03:06.873297 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:03:06.873266 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd999e2e2_910d_4c23_b784_31b2836a9c20.slice/crio-5ad48be3c6909f4f9d14926538dc7dc678c391ed7b24884056ab295aa5bf4301 WatchSource:0}: Error finding container 5ad48be3c6909f4f9d14926538dc7dc678c391ed7b24884056ab295aa5bf4301: Status 404 returned error can't find the container with id 5ad48be3c6909f4f9d14926538dc7dc678c391ed7b24884056ab295aa5bf4301 Nov 02 23:03:07.205764 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:07.205728 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2" event={"ID":"d999e2e2-910d-4c23-b784-31b2836a9c20","Type":"ContainerStarted","Data":"5ad48be3c6909f4f9d14926538dc7dc678c391ed7b24884056ab295aa5bf4301"} Nov 02 23:03:08.169863 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:08.169832 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-16aef2f3-794798c447-zpcwj"] Nov 02 23:03:08.172818 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:08.172795 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-16aef2f3-794798c447-zpcwj" Nov 02 23:03:08.181931 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:08.181840 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xknvg\" (UniqueName: \"kubernetes.io/projected/d0f6b370-0c0e-45a5-b612-c8a1b3036dc2-kube-api-access-xknvg\") pod \"payload-pre-processing-e2e-ait-16aef2f3-794798c447-zpcwj\" (UID: \"d0f6b370-0c0e-45a5-b612-c8a1b3036dc2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-16aef2f3-794798c447-zpcwj" Nov 02 23:03:08.181931 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:08.181999 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0f6b370-0c0e-45a5-b612-c8a1b3036dc2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-16aef2f3-794798c447-zpcwj\" (UID: \"d0f6b370-0c0e-45a5-b612-c8a1b3036dc2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-16aef2f3-794798c447-zpcwj" Nov 02 23:03:08.213964 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:08.213924 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8a68c823-d654c98dc-7v4lm" event={"ID":"dfebc122-0b39-4c69-aa55-8d4aafa72905","Type":"ContainerStarted","Data":"8e34f0fdef6889db57775205f66813acc1c27876e4f842c6e09085701e471973"} Nov 02 23:03:08.214148 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:08.214094 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-8a68c823-d654c98dc-7v4lm" Nov 02 23:03:08.216004 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:08.215976 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8a68c823-cfbd576f9-ffck9" event={"ID":"061b5c4a-b850-433e-ba62-3ae8de255048","Type":"ContainerStarted","Data":"fb38ddca83afb89e6ea40a6519f0d31b3f5ea12e0b4cbfbaf87e2762af3e8dd1"} Nov 02 23:03:08.216137 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:08.216116 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8a68c823-cfbd576f9-ffck9" Nov 02 23:03:08.284355 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:08.283618 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0f6b370-0c0e-45a5-b612-c8a1b3036dc2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-16aef2f3-794798c447-zpcwj\" (UID: \"d0f6b370-0c0e-45a5-b612-c8a1b3036dc2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-16aef2f3-794798c447-zpcwj" Nov 02 23:03:08.284355 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:08.283739 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xknvg\" (UniqueName: \"kubernetes.io/projected/d0f6b370-0c0e-45a5-b612-c8a1b3036dc2-kube-api-access-xknvg\") pod \"payload-pre-processing-e2e-ait-16aef2f3-794798c447-zpcwj\" (UID: \"d0f6b370-0c0e-45a5-b612-c8a1b3036dc2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-16aef2f3-794798c447-zpcwj" Nov 02 23:03:08.284616 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:08.284540 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-16aef2f3\"" Nov 02 23:03:08.284616 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:08.284564 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-16aef2f3-dockercfg-k6h2x\"" Nov 02 23:03:08.286692 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:08.286665 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-16aef2f3-794798c447-zpcwj"] Nov 02 23:03:08.295950 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:08.295922 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0f6b370-0c0e-45a5-b612-c8a1b3036dc2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-16aef2f3-794798c447-zpcwj\" (UID: \"d0f6b370-0c0e-45a5-b612-c8a1b3036dc2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-16aef2f3-794798c447-zpcwj" Nov 02 23:03:08.330744 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:08.330712 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-16aef2f3-7649b9bf5f-8ngg9"] Nov 02 23:03:08.334012 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:08.333974 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-16aef2f3-7649b9bf5f-8ngg9" Nov 02 23:03:08.341582 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:08.341553 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xknvg\" (UniqueName: \"kubernetes.io/projected/d0f6b370-0c0e-45a5-b612-c8a1b3036dc2-kube-api-access-xknvg\") pod \"payload-pre-processing-e2e-ait-16aef2f3-794798c447-zpcwj\" (UID: \"d0f6b370-0c0e-45a5-b612-c8a1b3036dc2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-16aef2f3-794798c447-zpcwj" Nov 02 23:03:08.385466 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:08.384570 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmj4b\" (UniqueName: \"kubernetes.io/projected/27c66990-66df-4a66-b3ec-e66a0e8fd540-kube-api-access-bmj4b\") pod \"payload-processing-e2e-ait-16aef2f3-7649b9bf5f-8ngg9\" (UID: \"27c66990-66df-4a66-b3ec-e66a0e8fd540\") " pod="openshift-ingress/payload-processing-e2e-ait-16aef2f3-7649b9bf5f-8ngg9" Nov 02 23:03:08.385466 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:08.384970 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c66990-66df-4a66-b3ec-e66a0e8fd540-plugins-config-volume\") pod \"payload-processing-e2e-ait-16aef2f3-7649b9bf5f-8ngg9\" (UID: \"27c66990-66df-4a66-b3ec-e66a0e8fd540\") " pod="openshift-ingress/payload-processing-e2e-ait-16aef2f3-7649b9bf5f-8ngg9" Nov 02 23:03:08.388465 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:08.388441 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-16aef2f3-7649b9bf5f-8ngg9"] Nov 02 23:03:08.454871 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:08.454817 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-8a68c823-d654c98dc-7v4lm" podStartSLOduration=2.386141761 podStartE2EDuration="5.45480243s" totalImagesPullingTime="3.068660669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:03:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:03:08.453432961 +0000 UTC m=+637.421964745" watchObservedRunningTime="2026-11-02 23:03:08.45480243 +0000 UTC m=+637.423334213" Nov 02 23:03:08.486902 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:08.486852 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c66990-66df-4a66-b3ec-e66a0e8fd540-plugins-config-volume\") pod \"payload-processing-e2e-ait-16aef2f3-7649b9bf5f-8ngg9\" (UID: \"27c66990-66df-4a66-b3ec-e66a0e8fd540\") " pod="openshift-ingress/payload-processing-e2e-ait-16aef2f3-7649b9bf5f-8ngg9" Nov 02 23:03:08.487113 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:08.487093 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmj4b\" (UniqueName: \"kubernetes.io/projected/27c66990-66df-4a66-b3ec-e66a0e8fd540-kube-api-access-bmj4b\") pod \"payload-processing-e2e-ait-16aef2f3-7649b9bf5f-8ngg9\" (UID: \"27c66990-66df-4a66-b3ec-e66a0e8fd540\") " pod="openshift-ingress/payload-processing-e2e-ait-16aef2f3-7649b9bf5f-8ngg9" Nov 02 23:03:08.487766 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:08.487740 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c66990-66df-4a66-b3ec-e66a0e8fd540-plugins-config-volume\") pod \"payload-processing-e2e-ait-16aef2f3-7649b9bf5f-8ngg9\" (UID: \"27c66990-66df-4a66-b3ec-e66a0e8fd540\") " pod="openshift-ingress/payload-processing-e2e-ait-16aef2f3-7649b9bf5f-8ngg9" Nov 02 23:03:08.489695 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:08.489671 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-16aef2f3-794798c447-zpcwj" Nov 02 23:03:08.542400 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:08.542365 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmj4b\" (UniqueName: \"kubernetes.io/projected/27c66990-66df-4a66-b3ec-e66a0e8fd540-kube-api-access-bmj4b\") pod \"payload-processing-e2e-ait-16aef2f3-7649b9bf5f-8ngg9\" (UID: \"27c66990-66df-4a66-b3ec-e66a0e8fd540\") " pod="openshift-ingress/payload-processing-e2e-ait-16aef2f3-7649b9bf5f-8ngg9" Nov 02 23:03:08.662722 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:08.662688 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-16aef2f3-7649b9bf5f-8ngg9" Nov 02 23:03:08.865700 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:08.865647 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8a68c823-cfbd576f9-ffck9" podStartSLOduration=2.719382487 podStartE2EDuration="5.865630719s" totalImagesPullingTime="3.146248232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:03:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:03:08.674187454 +0000 UTC m=+637.642719240" watchObservedRunningTime="2026-11-02 23:03:08.865630719 +0000 UTC m=+637.834162503" Nov 02 23:03:08.871200 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:08.869236 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-16aef2f3-794798c447-zpcwj"] Nov 02 23:03:08.884957 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:03:08.872497 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0f6b370_0c0e_45a5_b612_c8a1b3036dc2.slice/crio-a24d673f0a1f5af8b86e447275c461b045c5ee5b5c3e01dc77c77a2221604f7d WatchSource:0}: Error finding container a24d673f0a1f5af8b86e447275c461b045c5ee5b5c3e01dc77c77a2221604f7d: Status 404 returned error can't find the container with id a24d673f0a1f5af8b86e447275c461b045c5ee5b5c3e01dc77c77a2221604f7d Nov 02 23:03:09.044343 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:09.044314 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-16aef2f3-7649b9bf5f-8ngg9"] Nov 02 23:03:09.048033 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:03:09.047999 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27c66990_66df_4a66_b3ec_e66a0e8fd540.slice/crio-e6d1d7c6c086d929c38206be500c3b848c00943b35db1d3fca43f348c7e72f5f WatchSource:0}: Error finding container e6d1d7c6c086d929c38206be500c3b848c00943b35db1d3fca43f348c7e72f5f: Status 404 returned error can't find the container with id e6d1d7c6c086d929c38206be500c3b848c00943b35db1d3fca43f348c7e72f5f Nov 02 23:03:09.223747 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:09.223704 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-16aef2f3-794798c447-zpcwj" event={"ID":"d0f6b370-0c0e-45a5-b612-c8a1b3036dc2","Type":"ContainerStarted","Data":"2521652d19181a45dfbc2793d66400326bc4d12d70930730d2b3f7e3f5d6d671"} Nov 02 23:03:09.224222 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:09.223778 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-16aef2f3-794798c447-zpcwj" event={"ID":"d0f6b370-0c0e-45a5-b612-c8a1b3036dc2","Type":"ContainerStarted","Data":"a24d673f0a1f5af8b86e447275c461b045c5ee5b5c3e01dc77c77a2221604f7d"} Nov 02 23:03:09.224222 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:09.223809 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-16aef2f3-794798c447-zpcwj" Nov 02 23:03:09.225727 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:09.225701 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-16aef2f3-7649b9bf5f-8ngg9" event={"ID":"27c66990-66df-4a66-b3ec-e66a0e8fd540","Type":"ContainerStarted","Data":"fee83b119819b17f9280496dcc1f7b9f8d7c615bfb9d58d1b8cc12225d61b2e4"} Nov 02 23:03:09.225849 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:09.225753 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-16aef2f3-7649b9bf5f-8ngg9" event={"ID":"27c66990-66df-4a66-b3ec-e66a0e8fd540","Type":"ContainerStarted","Data":"e6d1d7c6c086d929c38206be500c3b848c00943b35db1d3fca43f348c7e72f5f"} Nov 02 23:03:09.226070 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:09.226048 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-16aef2f3-7649b9bf5f-8ngg9" Nov 02 23:03:09.332000 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:09.331860 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-16aef2f3-794798c447-zpcwj" podStartSLOduration=1.3318401 podStartE2EDuration="1.3318401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:03:09.33098741 +0000 UTC m=+638.299519215" watchObservedRunningTime="2026-11-02 23:03:09.3318401 +0000 UTC m=+638.300371888" Nov 02 23:03:09.438491 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:09.438429 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-16aef2f3-7649b9bf5f-8ngg9" podStartSLOduration=1.43841124 podStartE2EDuration="1.43841124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:03:09.438271995 +0000 UTC m=+638.406803779" watchObservedRunningTime="2026-11-02 23:03:09.43841124 +0000 UTC m=+638.406943023" Nov 02 23:03:09.887972 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:03:09.887932 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:03:10.491293 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:10.491207 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-db3e5cc4-6cb868b447-8fwl7"] Nov 02 23:03:10.494108 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:10.494087 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-db3e5cc4-6cb868b447-8fwl7" Nov 02 23:03:10.509823 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:10.509799 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kfg8\" (UniqueName: \"kubernetes.io/projected/e8efd632-03bb-452f-8da7-ffcf71621710-kube-api-access-2kfg8\") pod \"payload-pre-processing-e2e-ait-db3e5cc4-6cb868b447-8fwl7\" (UID: \"e8efd632-03bb-452f-8da7-ffcf71621710\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-db3e5cc4-6cb868b447-8fwl7" Nov 02 23:03:10.509971 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:10.509848 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8efd632-03bb-452f-8da7-ffcf71621710-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-db3e5cc4-6cb868b447-8fwl7\" (UID: \"e8efd632-03bb-452f-8da7-ffcf71621710\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-db3e5cc4-6cb868b447-8fwl7" Nov 02 23:03:10.545360 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:10.545288 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-db3e5cc4\"" Nov 02 23:03:10.599254 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:10.599203 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-db3e5cc4-6cb868b447-8fwl7"] Nov 02 23:03:10.599432 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:10.599414 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-db3e5cc4-dockercfg-mpbx6\"" Nov 02 23:03:10.611182 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:10.611150 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kfg8\" (UniqueName: \"kubernetes.io/projected/e8efd632-03bb-452f-8da7-ffcf71621710-kube-api-access-2kfg8\") pod \"payload-pre-processing-e2e-ait-db3e5cc4-6cb868b447-8fwl7\" (UID: \"e8efd632-03bb-452f-8da7-ffcf71621710\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-db3e5cc4-6cb868b447-8fwl7" Nov 02 23:03:10.612077 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:10.611239 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8efd632-03bb-452f-8da7-ffcf71621710-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-db3e5cc4-6cb868b447-8fwl7\" (UID: \"e8efd632-03bb-452f-8da7-ffcf71621710\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-db3e5cc4-6cb868b447-8fwl7" Nov 02 23:03:10.612242 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:10.612216 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8efd632-03bb-452f-8da7-ffcf71621710-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-db3e5cc4-6cb868b447-8fwl7\" (UID: \"e8efd632-03bb-452f-8da7-ffcf71621710\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-db3e5cc4-6cb868b447-8fwl7" Nov 02 23:03:10.647411 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:10.647375 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-db3e5cc4-698f549c7c-cjbdf"] Nov 02 23:03:10.649982 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:10.649962 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-db3e5cc4-698f549c7c-cjbdf" Nov 02 23:03:10.666837 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:10.666813 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kfg8\" (UniqueName: \"kubernetes.io/projected/e8efd632-03bb-452f-8da7-ffcf71621710-kube-api-access-2kfg8\") pod \"payload-pre-processing-e2e-ait-db3e5cc4-6cb868b447-8fwl7\" (UID: \"e8efd632-03bb-452f-8da7-ffcf71621710\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-db3e5cc4-6cb868b447-8fwl7" Nov 02 23:03:10.701090 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:10.701058 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-db3e5cc4-698f549c7c-cjbdf"] Nov 02 23:03:10.711801 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:10.711773 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66zx2\" (UniqueName: \"kubernetes.io/projected/a0807844-8cec-421b-b197-e1865248f554-kube-api-access-66zx2\") pod \"payload-processing-e2e-ait-db3e5cc4-698f549c7c-cjbdf\" (UID: \"a0807844-8cec-421b-b197-e1865248f554\") " pod="openshift-ingress/payload-processing-e2e-ait-db3e5cc4-698f549c7c-cjbdf" Nov 02 23:03:10.711959 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:10.711829 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0807844-8cec-421b-b197-e1865248f554-plugins-config-volume\") pod \"payload-processing-e2e-ait-db3e5cc4-698f549c7c-cjbdf\" (UID: \"a0807844-8cec-421b-b197-e1865248f554\") " pod="openshift-ingress/payload-processing-e2e-ait-db3e5cc4-698f549c7c-cjbdf" Nov 02 23:03:10.806846 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:10.806765 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-db3e5cc4-6cb868b447-8fwl7" Nov 02 23:03:10.813141 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:10.813119 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0807844-8cec-421b-b197-e1865248f554-plugins-config-volume\") pod \"payload-processing-e2e-ait-db3e5cc4-698f549c7c-cjbdf\" (UID: \"a0807844-8cec-421b-b197-e1865248f554\") " pod="openshift-ingress/payload-processing-e2e-ait-db3e5cc4-698f549c7c-cjbdf" Nov 02 23:03:10.813223 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:10.813207 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66zx2\" (UniqueName: \"kubernetes.io/projected/a0807844-8cec-421b-b197-e1865248f554-kube-api-access-66zx2\") pod \"payload-processing-e2e-ait-db3e5cc4-698f549c7c-cjbdf\" (UID: \"a0807844-8cec-421b-b197-e1865248f554\") " pod="openshift-ingress/payload-processing-e2e-ait-db3e5cc4-698f549c7c-cjbdf" Nov 02 23:03:10.814301 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:10.814277 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0807844-8cec-421b-b197-e1865248f554-plugins-config-volume\") pod \"payload-processing-e2e-ait-db3e5cc4-698f549c7c-cjbdf\" (UID: \"a0807844-8cec-421b-b197-e1865248f554\") " pod="openshift-ingress/payload-processing-e2e-ait-db3e5cc4-698f549c7c-cjbdf" Nov 02 23:03:10.868996 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:10.868931 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66zx2\" (UniqueName: \"kubernetes.io/projected/a0807844-8cec-421b-b197-e1865248f554-kube-api-access-66zx2\") pod \"payload-processing-e2e-ait-db3e5cc4-698f549c7c-cjbdf\" (UID: \"a0807844-8cec-421b-b197-e1865248f554\") " pod="openshift-ingress/payload-processing-e2e-ait-db3e5cc4-698f549c7c-cjbdf" Nov 02 23:03:10.962057 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:10.962025 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-db3e5cc4-698f549c7c-cjbdf" Nov 02 23:03:11.168582 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:11.168553 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-db3e5cc4-6cb868b447-8fwl7"] Nov 02 23:03:11.170108 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:03:11.170082 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8efd632_03bb_452f_8da7_ffcf71621710.slice/crio-ea6d474c23c1e6a34af72682c8b64e04f98519fac7849cbe2d85c4bd775bfe81 WatchSource:0}: Error finding container ea6d474c23c1e6a34af72682c8b64e04f98519fac7849cbe2d85c4bd775bfe81: Status 404 returned error can't find the container with id ea6d474c23c1e6a34af72682c8b64e04f98519fac7849cbe2d85c4bd775bfe81 Nov 02 23:03:11.240109 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:11.240076 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2" event={"ID":"d999e2e2-910d-4c23-b784-31b2836a9c20","Type":"ContainerStarted","Data":"9e7f8afaeb949738a595a1eaa9d1571f0d66e2f34a06ac29d53a6053a3d88801"} Nov 02 23:03:11.240242 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:11.240195 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2" Nov 02 23:03:11.241944 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:11.241918 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-db3e5cc4-6cb868b447-8fwl7" event={"ID":"e8efd632-03bb-452f-8da7-ffcf71621710","Type":"ContainerStarted","Data":"1254a90f822b92dc2291445daa4b9f53b5f5d3ed26a8e1f049d277d24f62c9f0"} Nov 02 23:03:11.242058 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:11.241959 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-db3e5cc4-6cb868b447-8fwl7" event={"ID":"e8efd632-03bb-452f-8da7-ffcf71621710","Type":"ContainerStarted","Data":"ea6d474c23c1e6a34af72682c8b64e04f98519fac7849cbe2d85c4bd775bfe81"} Nov 02 23:03:11.242058 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:11.241981 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-db3e5cc4-6cb868b447-8fwl7" Nov 02 23:03:11.329463 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:11.329432 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-db3e5cc4-698f549c7c-cjbdf"] Nov 02 23:03:11.330752 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:03:11.330727 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0807844_8cec_421b_b197_e1865248f554.slice/crio-cf4d6edcbe231966afda22ad39300394f038ac67be2d9ab5db15d8613e5de98a WatchSource:0}: Error finding container cf4d6edcbe231966afda22ad39300394f038ac67be2d9ab5db15d8613e5de98a: Status 404 returned error can't find the container with id cf4d6edcbe231966afda22ad39300394f038ac67be2d9ab5db15d8613e5de98a Nov 02 23:03:11.354115 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:11.354072 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2" podStartSLOduration=3.011975377 podStartE2EDuration="6.354057556s" totalImagesPullingTime="3.342082179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:03:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:03:11.353023389 +0000 UTC m=+640.321555173" watchObservedRunningTime="2026-11-02 23:03:11.354057556 +0000 UTC m=+640.322589339" Nov 02 23:03:11.477376 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:11.477313 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-db3e5cc4-6cb868b447-8fwl7" podStartSLOduration=1.4772921 podStartE2EDuration="1.4772921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:03:11.475032531 +0000 UTC m=+640.443564315" watchObservedRunningTime="2026-11-02 23:03:11.4772921 +0000 UTC m=+640.445823884" Nov 02 23:03:12.249876 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:12.249839 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-db3e5cc4-698f549c7c-cjbdf" event={"ID":"a0807844-8cec-421b-b197-e1865248f554","Type":"ContainerStarted","Data":"543c66524c2314c7da6298d7c03fbe93dfff8896ee6c4f64b6350f2375f0171e"} Nov 02 23:03:12.250345 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:12.249914 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-db3e5cc4-698f549c7c-cjbdf" event={"ID":"a0807844-8cec-421b-b197-e1865248f554","Type":"ContainerStarted","Data":"cf4d6edcbe231966afda22ad39300394f038ac67be2d9ab5db15d8613e5de98a"} Nov 02 23:03:12.250345 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:12.250139 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-db3e5cc4-698f549c7c-cjbdf" Nov 02 23:03:12.358094 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:12.358039 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-db3e5cc4-698f549c7c-cjbdf" podStartSLOduration=2.358024204 podStartE2EDuration="2.358024204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:03:12.357489836 +0000 UTC m=+641.326021621" watchObservedRunningTime="2026-11-02 23:03:12.358024204 +0000 UTC m=+641.326555988" Nov 02 23:03:15.155819 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:15.155786 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d0078993-6cd56965b7-5t6bm"] Nov 02 23:03:15.158442 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:15.158427 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0078993-6cd56965b7-5t6bm" Nov 02 23:03:15.214919 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:15.214849 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d0078993\"" Nov 02 23:03:15.262328 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:15.262298 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d0078993-dockercfg-9q2z9\"" Nov 02 23:03:15.262643 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:15.262610 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzxzq\" (UniqueName: \"kubernetes.io/projected/5de5725b-cf19-4e08-8a28-cc2be38ff76f-kube-api-access-mzxzq\") pod \"payload-pre-processing-e2e-ait-d0078993-6cd56965b7-5t6bm\" (UID: \"5de5725b-cf19-4e08-8a28-cc2be38ff76f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d0078993-6cd56965b7-5t6bm" Nov 02 23:03:15.262728 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:15.262671 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5de5725b-cf19-4e08-8a28-cc2be38ff76f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d0078993-6cd56965b7-5t6bm\" (UID: \"5de5725b-cf19-4e08-8a28-cc2be38ff76f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d0078993-6cd56965b7-5t6bm" Nov 02 23:03:15.264707 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:15.264685 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d0078993-6cd56965b7-5t6bm"] Nov 02 23:03:15.313833 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:15.313802 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d0078993-57c5855b4c-htk5d"] Nov 02 23:03:15.316468 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:15.316452 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d0078993-57c5855b4c-htk5d" Nov 02 23:03:15.363816 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:15.363779 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzxzq\" (UniqueName: \"kubernetes.io/projected/5de5725b-cf19-4e08-8a28-cc2be38ff76f-kube-api-access-mzxzq\") pod \"payload-pre-processing-e2e-ait-d0078993-6cd56965b7-5t6bm\" (UID: \"5de5725b-cf19-4e08-8a28-cc2be38ff76f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d0078993-6cd56965b7-5t6bm" Nov 02 23:03:15.363999 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:15.363831 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5de5725b-cf19-4e08-8a28-cc2be38ff76f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d0078993-6cd56965b7-5t6bm\" (UID: \"5de5725b-cf19-4e08-8a28-cc2be38ff76f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d0078993-6cd56965b7-5t6bm" Nov 02 23:03:15.363999 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:15.363883 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f359fa32-c147-4ddd-8b34-709bec74a203-plugins-config-volume\") pod \"payload-processing-e2e-ait-d0078993-57c5855b4c-htk5d\" (UID: \"f359fa32-c147-4ddd-8b34-709bec74a203\") " pod="openshift-ingress/payload-processing-e2e-ait-d0078993-57c5855b4c-htk5d" Nov 02 23:03:15.364090 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:15.364043 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpf55\" (UniqueName: \"kubernetes.io/projected/f359fa32-c147-4ddd-8b34-709bec74a203-kube-api-access-rpf55\") pod \"payload-processing-e2e-ait-d0078993-57c5855b4c-htk5d\" (UID: \"f359fa32-c147-4ddd-8b34-709bec74a203\") " pod="openshift-ingress/payload-processing-e2e-ait-d0078993-57c5855b4c-htk5d" Nov 02 23:03:15.364582 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:15.364565 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5de5725b-cf19-4e08-8a28-cc2be38ff76f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d0078993-6cd56965b7-5t6bm\" (UID: \"5de5725b-cf19-4e08-8a28-cc2be38ff76f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d0078993-6cd56965b7-5t6bm" Nov 02 23:03:15.369302 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:15.369275 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d0078993-57c5855b4c-htk5d"] Nov 02 23:03:15.418309 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:15.418233 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzxzq\" (UniqueName: \"kubernetes.io/projected/5de5725b-cf19-4e08-8a28-cc2be38ff76f-kube-api-access-mzxzq\") pod \"payload-pre-processing-e2e-ait-d0078993-6cd56965b7-5t6bm\" (UID: \"5de5725b-cf19-4e08-8a28-cc2be38ff76f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d0078993-6cd56965b7-5t6bm" Nov 02 23:03:15.465207 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:15.465171 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f359fa32-c147-4ddd-8b34-709bec74a203-plugins-config-volume\") pod \"payload-processing-e2e-ait-d0078993-57c5855b4c-htk5d\" (UID: \"f359fa32-c147-4ddd-8b34-709bec74a203\") " pod="openshift-ingress/payload-processing-e2e-ait-d0078993-57c5855b4c-htk5d" Nov 02 23:03:15.465207 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:15.465210 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpf55\" (UniqueName: \"kubernetes.io/projected/f359fa32-c147-4ddd-8b34-709bec74a203-kube-api-access-rpf55\") pod \"payload-processing-e2e-ait-d0078993-57c5855b4c-htk5d\" (UID: \"f359fa32-c147-4ddd-8b34-709bec74a203\") " pod="openshift-ingress/payload-processing-e2e-ait-d0078993-57c5855b4c-htk5d" Nov 02 23:03:15.465928 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:15.465886 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f359fa32-c147-4ddd-8b34-709bec74a203-plugins-config-volume\") pod \"payload-processing-e2e-ait-d0078993-57c5855b4c-htk5d\" (UID: \"f359fa32-c147-4ddd-8b34-709bec74a203\") " pod="openshift-ingress/payload-processing-e2e-ait-d0078993-57c5855b4c-htk5d" Nov 02 23:03:15.470032 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:15.470012 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0078993-6cd56965b7-5t6bm" Nov 02 23:03:15.521466 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:15.519925 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpf55\" (UniqueName: \"kubernetes.io/projected/f359fa32-c147-4ddd-8b34-709bec74a203-kube-api-access-rpf55\") pod \"payload-processing-e2e-ait-d0078993-57c5855b4c-htk5d\" (UID: \"f359fa32-c147-4ddd-8b34-709bec74a203\") " pod="openshift-ingress/payload-processing-e2e-ait-d0078993-57c5855b4c-htk5d" Nov 02 23:03:15.628709 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:15.628676 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d0078993-57c5855b4c-htk5d" Nov 02 23:03:15.835283 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:15.835256 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d0078993-6cd56965b7-5t6bm"] Nov 02 23:03:15.835979 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:03:15.835954 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5de5725b_cf19_4e08_8a28_cc2be38ff76f.slice/crio-3d77e9ae697bfc78af8a7d9ca2b252f7568a331a2ebf58fd4200dc0d78820ece WatchSource:0}: Error finding container 3d77e9ae697bfc78af8a7d9ca2b252f7568a331a2ebf58fd4200dc0d78820ece: Status 404 returned error can't find the container with id 3d77e9ae697bfc78af8a7d9ca2b252f7568a331a2ebf58fd4200dc0d78820ece Nov 02 23:03:15.990033 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:15.990004 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d0078993-57c5855b4c-htk5d"] Nov 02 23:03:15.991219 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:03:15.991194 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf359fa32_c147_4ddd_8b34_709bec74a203.slice/crio-e732651cf10816a5e761effdfb33ac7a1dba1cb158b6e31e0f54dc2448dc6197 WatchSource:0}: Error finding container e732651cf10816a5e761effdfb33ac7a1dba1cb158b6e31e0f54dc2448dc6197: Status 404 returned error can't find the container with id e732651cf10816a5e761effdfb33ac7a1dba1cb158b6e31e0f54dc2448dc6197 Nov 02 23:03:16.272155 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:16.272064 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0078993-6cd56965b7-5t6bm" event={"ID":"5de5725b-cf19-4e08-8a28-cc2be38ff76f","Type":"ContainerStarted","Data":"f960e5e88ce2a4f0f92530f6e3df10cb9ac49c4f74997e3f04f41f7342096a16"} Nov 02 23:03:16.272155 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:16.272114 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0078993-6cd56965b7-5t6bm" event={"ID":"5de5725b-cf19-4e08-8a28-cc2be38ff76f","Type":"ContainerStarted","Data":"3d77e9ae697bfc78af8a7d9ca2b252f7568a331a2ebf58fd4200dc0d78820ece"} Nov 02 23:03:16.272621 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:16.272248 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0078993-6cd56965b7-5t6bm" Nov 02 23:03:16.273711 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:16.273689 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d0078993-57c5855b4c-htk5d" event={"ID":"f359fa32-c147-4ddd-8b34-709bec74a203","Type":"ContainerStarted","Data":"64a2ad11044456abb9c53fa3d6df9f9a47bb2436ea83edd0ac3654e96379793e"} Nov 02 23:03:16.273844 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:16.273724 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d0078993-57c5855b4c-htk5d" event={"ID":"f359fa32-c147-4ddd-8b34-709bec74a203","Type":"ContainerStarted","Data":"e732651cf10816a5e761effdfb33ac7a1dba1cb158b6e31e0f54dc2448dc6197"} Nov 02 23:03:16.273844 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:16.273818 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d0078993-57c5855b4c-htk5d" Nov 02 23:03:16.380825 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:16.380777 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0078993-6cd56965b7-5t6bm" podStartSLOduration=1.380763269 podStartE2EDuration="1.380763269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:03:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:03:16.379288366 +0000 UTC m=+645.347820150" watchObservedRunningTime="2026-11-02 23:03:16.380763269 +0000 UTC m=+645.349295052" Nov 02 23:03:16.489684 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:16.489618 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d0078993-57c5855b4c-htk5d" podStartSLOduration=1.489598566 podStartE2EDuration="1.489598566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:03:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:03:16.486929255 +0000 UTC m=+645.455461041" watchObservedRunningTime="2026-11-02 23:03:16.489598566 +0000 UTC m=+645.458130351" Nov 02 23:03:17.256204 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:17.256169 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2" Nov 02 23:03:19.229735 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:19.229705 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-8a68c823-d654c98dc-7v4lm" Nov 02 23:03:19.230110 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:19.229773 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8a68c823-cfbd576f9-ffck9" Nov 02 23:03:20.236670 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:20.236639 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-16aef2f3-794798c447-zpcwj" Nov 02 23:03:21.245318 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:21.245290 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-16aef2f3-7649b9bf5f-8ngg9" Nov 02 23:03:22.064801 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:22.064765 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bdaf93e3-57cf9555-cc2rb"] Nov 02 23:03:22.067872 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:22.067854 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bdaf93e3-57cf9555-cc2rb" Nov 02 23:03:22.122528 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:22.122495 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bdaf93e3-57cf9555-cc2rb"] Nov 02 23:03:22.152999 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:22.152959 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b1d58ec-753e-46e8-882d-b0c3fe6f91cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-bdaf93e3-57cf9555-cc2rb\" (UID: \"6b1d58ec-753e-46e8-882d-b0c3fe6f91cc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bdaf93e3-57cf9555-cc2rb" Nov 02 23:03:22.153212 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:22.153178 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4v29\" (UniqueName: \"kubernetes.io/projected/6b1d58ec-753e-46e8-882d-b0c3fe6f91cc-kube-api-access-z4v29\") pod \"payload-pre-processing-e2e-derive-bdaf93e3-57cf9555-cc2rb\" (UID: \"6b1d58ec-753e-46e8-882d-b0c3fe6f91cc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bdaf93e3-57cf9555-cc2rb" Nov 02 23:03:22.167696 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:22.167499 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-bdaf93e3-dockercfg-8tws5\"" Nov 02 23:03:22.167696 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:22.167585 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-bdaf93e3\"" Nov 02 23:03:22.190084 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:22.190055 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bdaf93e3-7446956765-qf8nk"] Nov 02 23:03:22.192762 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:22.192745 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-bdaf93e3-7446956765-qf8nk" Nov 02 23:03:22.247370 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:22.247329 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bdaf93e3-7446956765-qf8nk"] Nov 02 23:03:22.253480 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:22.253454 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-db3e5cc4-6cb868b447-8fwl7" Nov 02 23:03:22.254165 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:22.254140 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b1d58ec-753e-46e8-882d-b0c3fe6f91cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-bdaf93e3-57cf9555-cc2rb\" (UID: \"6b1d58ec-753e-46e8-882d-b0c3fe6f91cc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bdaf93e3-57cf9555-cc2rb" Nov 02 23:03:22.254267 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:22.254185 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8rb5\" (UniqueName: \"kubernetes.io/projected/5769d8f9-955f-43b4-9f73-433e625aea56-kube-api-access-p8rb5\") pod \"payload-processing-e2e-derive-bdaf93e3-7446956765-qf8nk\" (UID: \"5769d8f9-955f-43b4-9f73-433e625aea56\") " pod="openshift-ingress/payload-processing-e2e-derive-bdaf93e3-7446956765-qf8nk" Nov 02 23:03:22.254267 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:22.254246 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4v29\" (UniqueName: \"kubernetes.io/projected/6b1d58ec-753e-46e8-882d-b0c3fe6f91cc-kube-api-access-z4v29\") pod \"payload-pre-processing-e2e-derive-bdaf93e3-57cf9555-cc2rb\" (UID: \"6b1d58ec-753e-46e8-882d-b0c3fe6f91cc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bdaf93e3-57cf9555-cc2rb" Nov 02 23:03:22.254353 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:22.254304 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5769d8f9-955f-43b4-9f73-433e625aea56-plugins-config-volume\") pod \"payload-processing-e2e-derive-bdaf93e3-7446956765-qf8nk\" (UID: \"5769d8f9-955f-43b4-9f73-433e625aea56\") " pod="openshift-ingress/payload-processing-e2e-derive-bdaf93e3-7446956765-qf8nk" Nov 02 23:03:22.255266 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:22.255244 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b1d58ec-753e-46e8-882d-b0c3fe6f91cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-bdaf93e3-57cf9555-cc2rb\" (UID: \"6b1d58ec-753e-46e8-882d-b0c3fe6f91cc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bdaf93e3-57cf9555-cc2rb" Nov 02 23:03:22.308835 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:22.308805 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4v29\" (UniqueName: \"kubernetes.io/projected/6b1d58ec-753e-46e8-882d-b0c3fe6f91cc-kube-api-access-z4v29\") pod \"payload-pre-processing-e2e-derive-bdaf93e3-57cf9555-cc2rb\" (UID: \"6b1d58ec-753e-46e8-882d-b0c3fe6f91cc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bdaf93e3-57cf9555-cc2rb" Nov 02 23:03:22.360016 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:22.358628 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8rb5\" (UniqueName: \"kubernetes.io/projected/5769d8f9-955f-43b4-9f73-433e625aea56-kube-api-access-p8rb5\") pod \"payload-processing-e2e-derive-bdaf93e3-7446956765-qf8nk\" (UID: \"5769d8f9-955f-43b4-9f73-433e625aea56\") " pod="openshift-ingress/payload-processing-e2e-derive-bdaf93e3-7446956765-qf8nk" Nov 02 23:03:22.360016 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:22.359668 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5769d8f9-955f-43b4-9f73-433e625aea56-plugins-config-volume\") pod \"payload-processing-e2e-derive-bdaf93e3-7446956765-qf8nk\" (UID: \"5769d8f9-955f-43b4-9f73-433e625aea56\") " pod="openshift-ingress/payload-processing-e2e-derive-bdaf93e3-7446956765-qf8nk" Nov 02 23:03:22.360573 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:22.360555 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5769d8f9-955f-43b4-9f73-433e625aea56-plugins-config-volume\") pod \"payload-processing-e2e-derive-bdaf93e3-7446956765-qf8nk\" (UID: \"5769d8f9-955f-43b4-9f73-433e625aea56\") " pod="openshift-ingress/payload-processing-e2e-derive-bdaf93e3-7446956765-qf8nk" Nov 02 23:03:22.381030 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:22.381002 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bdaf93e3-57cf9555-cc2rb" Nov 02 23:03:22.414969 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:22.414935 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8rb5\" (UniqueName: \"kubernetes.io/projected/5769d8f9-955f-43b4-9f73-433e625aea56-kube-api-access-p8rb5\") pod \"payload-processing-e2e-derive-bdaf93e3-7446956765-qf8nk\" (UID: \"5769d8f9-955f-43b4-9f73-433e625aea56\") " pod="openshift-ingress/payload-processing-e2e-derive-bdaf93e3-7446956765-qf8nk" Nov 02 23:03:22.504979 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:22.504945 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-bdaf93e3-7446956765-qf8nk" Nov 02 23:03:22.756213 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:22.756185 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bdaf93e3-57cf9555-cc2rb"] Nov 02 23:03:22.757036 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:03:22.757004 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b1d58ec_753e_46e8_882d_b0c3fe6f91cc.slice/crio-119eb0af604bd14ce159ae058dc430b8400783d57e28176a5dda95c0b57fd49d WatchSource:0}: Error finding container 119eb0af604bd14ce159ae058dc430b8400783d57e28176a5dda95c0b57fd49d: Status 404 returned error can't find the container with id 119eb0af604bd14ce159ae058dc430b8400783d57e28176a5dda95c0b57fd49d Nov 02 23:03:22.878098 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:22.878063 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bdaf93e3-7446956765-qf8nk"] Nov 02 23:03:22.882248 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:03:22.882212 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5769d8f9_955f_43b4_9f73_433e625aea56.slice/crio-f054277b06d768de29149e9585f49bb17abf93b48ede7956036ebe1967227598 WatchSource:0}: Error finding container f054277b06d768de29149e9585f49bb17abf93b48ede7956036ebe1967227598: Status 404 returned error can't find the container with id f054277b06d768de29149e9585f49bb17abf93b48ede7956036ebe1967227598 Nov 02 23:03:23.258093 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:23.258062 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-db3e5cc4-698f549c7c-cjbdf" Nov 02 23:03:23.311743 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:23.311704 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bdaf93e3-57cf9555-cc2rb" event={"ID":"6b1d58ec-753e-46e8-882d-b0c3fe6f91cc","Type":"ContainerStarted","Data":"03a03c27394468acca6d7c358155304d6d8537017f7de32c647d215ea4c1ce36"} Nov 02 23:03:23.311929 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:23.311770 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bdaf93e3-57cf9555-cc2rb" event={"ID":"6b1d58ec-753e-46e8-882d-b0c3fe6f91cc","Type":"ContainerStarted","Data":"119eb0af604bd14ce159ae058dc430b8400783d57e28176a5dda95c0b57fd49d"} Nov 02 23:03:23.312437 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:23.312415 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-bdaf93e3-57cf9555-cc2rb" Nov 02 23:03:23.314568 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:23.314539 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-bdaf93e3-7446956765-qf8nk" event={"ID":"5769d8f9-955f-43b4-9f73-433e625aea56","Type":"ContainerStarted","Data":"8ba0019109705eefcc2d98025c23f8fd3e7cf8adfab6082241494e3c17e45201"} Nov 02 23:03:23.314702 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:23.314604 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-bdaf93e3-7446956765-qf8nk" Nov 02 23:03:23.314702 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:23.314630 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-bdaf93e3-7446956765-qf8nk" event={"ID":"5769d8f9-955f-43b4-9f73-433e625aea56","Type":"ContainerStarted","Data":"f054277b06d768de29149e9585f49bb17abf93b48ede7956036ebe1967227598"} Nov 02 23:03:23.473573 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:23.473515 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-bdaf93e3-57cf9555-cc2rb" podStartSLOduration=1.473502016 podStartE2EDuration="1.473502016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:03:23.473166323 +0000 UTC m=+652.441698105" watchObservedRunningTime="2026-11-02 23:03:23.473502016 +0000 UTC m=+652.442033799" Nov 02 23:03:23.593222 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:23.593164 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-bdaf93e3-7446956765-qf8nk" podStartSLOduration=1.5931460400000002 podStartE2EDuration="1.59314604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:03:23.590162018 +0000 UTC m=+652.558693820" watchObservedRunningTime="2026-11-02 23:03:23.59314604 +0000 UTC m=+652.561677823" Nov 02 23:03:27.280792 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:27.280756 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d0078993-57c5855b4c-htk5d" Nov 02 23:03:27.281185 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:27.280825 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0078993-6cd56965b7-5t6bm" Nov 02 23:03:34.323229 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:34.323197 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-bdaf93e3-7446956765-qf8nk" Nov 02 23:03:35.327398 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:35.327370 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-bdaf93e3-57cf9555-cc2rb" Nov 02 23:03:39.882779 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:03:39.882739 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:03:47.552171 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:47.552077 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8a68c823-d654c98dc-7v4lm"] Nov 02 23:03:47.658577 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:47.658537 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8a68c823-cfbd576f9-ffck9"] Nov 02 23:03:52.552679 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:52.552630 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-8a68c823-d654c98dc-7v4lm" podUID="dfebc122-0b39-4c69-aa55-8d4aafa72905" containerName="payload-processing" containerID="cri-o://8e34f0fdef6889db57775205f66813acc1c27876e4f842c6e09085701e471973" gracePeriod=25 Nov 02 23:03:52.659091 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:52.659036 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8a68c823-cfbd576f9-ffck9" podUID="061b5c4a-b850-433e-ba62-3ae8de255048" containerName="payload-pre-processing" containerID="cri-o://fb38ddca83afb89e6ea40a6519f0d31b3f5ea12e0b4cbfbaf87e2762af3e8dd1" gracePeriod=25 Nov 02 23:03:52.824722 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:52.824687 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8a68c823-d654c98dc-7v4lm" Nov 02 23:03:52.844081 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:52.843772 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfebc122-0b39-4c69-aa55-8d4aafa72905-kube-api-access-tvc5r\" (UniqueName: \"kubernetes.io/projected/dfebc122-0b39-4c69-aa55-8d4aafa72905-kube-api-access-tvc5r\") pod \"dfebc122-0b39-4c69-aa55-8d4aafa72905\" (UID: \"dfebc122-0b39-4c69-aa55-8d4aafa72905\") " Nov 02 23:03:52.844081 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:52.843829 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfebc122-0b39-4c69-aa55-8d4aafa72905-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfebc122-0b39-4c69-aa55-8d4aafa72905-plugins-config-volume\") pod \"dfebc122-0b39-4c69-aa55-8d4aafa72905\" (UID: \"dfebc122-0b39-4c69-aa55-8d4aafa72905\") " Nov 02 23:03:52.844307 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:52.844268 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfebc122-0b39-4c69-aa55-8d4aafa72905-plugins-config-volume" pod "dfebc122-0b39-4c69-aa55-8d4aafa72905" (UID: "dfebc122-0b39-4c69-aa55-8d4aafa72905"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:03:52.844785 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:52.844743 5076 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfebc122-0b39-4c69-aa55-8d4aafa72905-plugins-config-volume\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:03:52.847944 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:52.847917 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfebc122-0b39-4c69-aa55-8d4aafa72905-kube-api-access-tvc5r" pod "dfebc122-0b39-4c69-aa55-8d4aafa72905" (UID: "dfebc122-0b39-4c69-aa55-8d4aafa72905"). InnerVolumeSpecName "kube-api-access-tvc5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:03:52.939470 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:52.939441 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8a68c823-cfbd576f9-ffck9" Nov 02 23:03:52.945645 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:52.945614 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvc5r\" (UniqueName: \"kubernetes.io/projected/dfebc122-0b39-4c69-aa55-8d4aafa72905-kube-api-access-tvc5r\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:03:53.046287 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:53.046249 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/061b5c4a-b850-433e-ba62-3ae8de255048-kube-api-access-85jcz\" (UniqueName: \"kubernetes.io/projected/061b5c4a-b850-433e-ba62-3ae8de255048-kube-api-access-85jcz\") pod \"061b5c4a-b850-433e-ba62-3ae8de255048\" (UID: \"061b5c4a-b850-433e-ba62-3ae8de255048\") " Nov 02 23:03:53.046488 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:53.046310 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/061b5c4a-b850-433e-ba62-3ae8de255048-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/061b5c4a-b850-433e-ba62-3ae8de255048-plugins-config-volume\") pod \"061b5c4a-b850-433e-ba62-3ae8de255048\" (UID: \"061b5c4a-b850-433e-ba62-3ae8de255048\") " Nov 02 23:03:53.046709 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:53.046682 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/061b5c4a-b850-433e-ba62-3ae8de255048-plugins-config-volume" pod "061b5c4a-b850-433e-ba62-3ae8de255048" (UID: "061b5c4a-b850-433e-ba62-3ae8de255048"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:03:53.048824 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:53.048794 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/061b5c4a-b850-433e-ba62-3ae8de255048-kube-api-access-85jcz" pod "061b5c4a-b850-433e-ba62-3ae8de255048" (UID: "061b5c4a-b850-433e-ba62-3ae8de255048"). InnerVolumeSpecName "kube-api-access-85jcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:03:53.147716 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:53.147640 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85jcz\" (UniqueName: \"kubernetes.io/projected/061b5c4a-b850-433e-ba62-3ae8de255048-kube-api-access-85jcz\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:03:53.147716 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:53.147665 5076 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/061b5c4a-b850-433e-ba62-3ae8de255048-plugins-config-volume\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:03:53.492112 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:53.492072 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8a68c823-d654c98dc-7v4lm" event={"ID":"dfebc122-0b39-4c69-aa55-8d4aafa72905","Type":"ContainerDied","Data":"8e34f0fdef6889db57775205f66813acc1c27876e4f842c6e09085701e471973"} Nov 02 23:03:53.492292 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:53.492101 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8a68c823-d654c98dc-7v4lm" Nov 02 23:03:53.492487 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:53.492025 5076 generic.go:356] "Generic (PLEG): container finished" podID="dfebc122-0b39-4c69-aa55-8d4aafa72905" containerID="8e34f0fdef6889db57775205f66813acc1c27876e4f842c6e09085701e471973" exitCode=0 Nov 02 23:03:53.492551 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:53.492516 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8a68c823-d654c98dc-7v4lm" event={"ID":"dfebc122-0b39-4c69-aa55-8d4aafa72905","Type":"ContainerDied","Data":"bdc2e60f15cad5dcfd1dd1c4a9d6772499f7380059dfc0a302ab507dd81382ae"} Nov 02 23:03:53.492605 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:53.492560 5076 scope.go:122] "RemoveContainer" containerID="8e34f0fdef6889db57775205f66813acc1c27876e4f842c6e09085701e471973" Nov 02 23:03:53.494341 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:53.494311 5076 generic.go:356] "Generic (PLEG): container finished" podID="061b5c4a-b850-433e-ba62-3ae8de255048" containerID="fb38ddca83afb89e6ea40a6519f0d31b3f5ea12e0b4cbfbaf87e2762af3e8dd1" exitCode=0 Nov 02 23:03:53.494468 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:53.494350 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8a68c823-cfbd576f9-ffck9" event={"ID":"061b5c4a-b850-433e-ba62-3ae8de255048","Type":"ContainerDied","Data":"fb38ddca83afb89e6ea40a6519f0d31b3f5ea12e0b4cbfbaf87e2762af3e8dd1"} Nov 02 23:03:53.494468 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:53.494367 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8a68c823-cfbd576f9-ffck9" Nov 02 23:03:53.494468 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:53.494390 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8a68c823-cfbd576f9-ffck9" event={"ID":"061b5c4a-b850-433e-ba62-3ae8de255048","Type":"ContainerDied","Data":"9afe818b5adaa2e443a642dd023b2c96dced2b00ee4da2567a63717167257ed9"} Nov 02 23:03:53.501841 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:53.501822 5076 scope.go:122] "RemoveContainer" containerID="8e34f0fdef6889db57775205f66813acc1c27876e4f842c6e09085701e471973" Nov 02 23:03:53.502119 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:03:53.502097 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e34f0fdef6889db57775205f66813acc1c27876e4f842c6e09085701e471973\": container with ID starting with 8e34f0fdef6889db57775205f66813acc1c27876e4f842c6e09085701e471973 not found: ID does not exist" containerID="8e34f0fdef6889db57775205f66813acc1c27876e4f842c6e09085701e471973" Nov 02 23:03:53.502194 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:53.502127 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e34f0fdef6889db57775205f66813acc1c27876e4f842c6e09085701e471973"} err="failed to get container status \"8e34f0fdef6889db57775205f66813acc1c27876e4f842c6e09085701e471973\": rpc error: code = NotFound desc = could not find container \"8e34f0fdef6889db57775205f66813acc1c27876e4f842c6e09085701e471973\": container with ID starting with 8e34f0fdef6889db57775205f66813acc1c27876e4f842c6e09085701e471973 not found: ID does not exist" Nov 02 23:03:53.502194 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:53.502145 5076 scope.go:122] "RemoveContainer" containerID="fb38ddca83afb89e6ea40a6519f0d31b3f5ea12e0b4cbfbaf87e2762af3e8dd1" Nov 02 23:03:53.510106 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:53.510087 5076 scope.go:122] "RemoveContainer" containerID="fb38ddca83afb89e6ea40a6519f0d31b3f5ea12e0b4cbfbaf87e2762af3e8dd1" Nov 02 23:03:53.510348 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:03:53.510330 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb38ddca83afb89e6ea40a6519f0d31b3f5ea12e0b4cbfbaf87e2762af3e8dd1\": container with ID starting with fb38ddca83afb89e6ea40a6519f0d31b3f5ea12e0b4cbfbaf87e2762af3e8dd1 not found: ID does not exist" containerID="fb38ddca83afb89e6ea40a6519f0d31b3f5ea12e0b4cbfbaf87e2762af3e8dd1" Nov 02 23:03:53.510398 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:53.510356 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb38ddca83afb89e6ea40a6519f0d31b3f5ea12e0b4cbfbaf87e2762af3e8dd1"} err="failed to get container status \"fb38ddca83afb89e6ea40a6519f0d31b3f5ea12e0b4cbfbaf87e2762af3e8dd1\": rpc error: code = NotFound desc = could not find container \"fb38ddca83afb89e6ea40a6519f0d31b3f5ea12e0b4cbfbaf87e2762af3e8dd1\": container with ID starting with fb38ddca83afb89e6ea40a6519f0d31b3f5ea12e0b4cbfbaf87e2762af3e8dd1 not found: ID does not exist" Nov 02 23:03:53.648716 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:53.648683 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8a68c823-cfbd576f9-ffck9"] Nov 02 23:03:53.651088 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:53.651062 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8a68c823-cfbd576f9-ffck9"] Nov 02 23:03:53.756485 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:53.756401 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8a68c823-d654c98dc-7v4lm"] Nov 02 23:03:53.763676 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:53.763644 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8a68c823-d654c98dc-7v4lm"] Nov 02 23:03:55.146963 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:55.146932 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="061b5c4a-b850-433e-ba62-3ae8de255048" path="/var/lib/kubelet/pods/061b5c4a-b850-433e-ba62-3ae8de255048/volumes" Nov 02 23:03:55.147393 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:55.147322 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfebc122-0b39-4c69-aa55-8d4aafa72905" path="/var/lib/kubelet/pods/dfebc122-0b39-4c69-aa55-8d4aafa72905/volumes" Nov 02 23:03:57.867744 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:57.867710 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2"] Nov 02 23:03:57.868233 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:57.867945 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2" podUID="d999e2e2-910d-4c23-b784-31b2836a9c20" containerName="maas-api" containerID="cri-o://9e7f8afaeb949738a595a1eaa9d1571f0d66e2f34a06ac29d53a6053a3d88801" gracePeriod=30 Nov 02 23:03:58.144349 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:58.144323 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2" Nov 02 23:03:58.209010 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:58.208976 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d999e2e2-910d-4c23-b784-31b2836a9c20-kube-api-access-vbkqq\" (UniqueName: \"kubernetes.io/projected/d999e2e2-910d-4c23-b784-31b2836a9c20-kube-api-access-vbkqq\") pod \"d999e2e2-910d-4c23-b784-31b2836a9c20\" (UID: \"d999e2e2-910d-4c23-b784-31b2836a9c20\") " Nov 02 23:03:58.209186 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:58.209028 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d999e2e2-910d-4c23-b784-31b2836a9c20-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d999e2e2-910d-4c23-b784-31b2836a9c20-metrics-certs\") pod \"d999e2e2-910d-4c23-b784-31b2836a9c20\" (UID: \"d999e2e2-910d-4c23-b784-31b2836a9c20\") " Nov 02 23:03:58.209186 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:58.209110 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d999e2e2-910d-4c23-b784-31b2836a9c20-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d999e2e2-910d-4c23-b784-31b2836a9c20-openshift-service-ca\") pod \"d999e2e2-910d-4c23-b784-31b2836a9c20\" (UID: \"d999e2e2-910d-4c23-b784-31b2836a9c20\") " Nov 02 23:03:58.209186 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:58.209142 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d999e2e2-910d-4c23-b784-31b2836a9c20-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d999e2e2-910d-4c23-b784-31b2836a9c20-maas-api-tls\") pod \"d999e2e2-910d-4c23-b784-31b2836a9c20\" (UID: \"d999e2e2-910d-4c23-b784-31b2836a9c20\") " Nov 02 23:03:58.209494 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:58.209459 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d999e2e2-910d-4c23-b784-31b2836a9c20-openshift-service-ca" pod "d999e2e2-910d-4c23-b784-31b2836a9c20" (UID: "d999e2e2-910d-4c23-b784-31b2836a9c20"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:03:58.211297 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:58.211270 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d999e2e2-910d-4c23-b784-31b2836a9c20-metrics-certs" pod "d999e2e2-910d-4c23-b784-31b2836a9c20" (UID: "d999e2e2-910d-4c23-b784-31b2836a9c20"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:03:58.211397 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:58.211358 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d999e2e2-910d-4c23-b784-31b2836a9c20-kube-api-access-vbkqq" pod "d999e2e2-910d-4c23-b784-31b2836a9c20" (UID: "d999e2e2-910d-4c23-b784-31b2836a9c20"). InnerVolumeSpecName "kube-api-access-vbkqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:03:58.211447 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:58.211423 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d999e2e2-910d-4c23-b784-31b2836a9c20-maas-api-tls" pod "d999e2e2-910d-4c23-b784-31b2836a9c20" (UID: "d999e2e2-910d-4c23-b784-31b2836a9c20"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:03:58.310554 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:58.310519 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbkqq\" (UniqueName: \"kubernetes.io/projected/d999e2e2-910d-4c23-b784-31b2836a9c20-kube-api-access-vbkqq\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:03:58.310554 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:58.310548 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d999e2e2-910d-4c23-b784-31b2836a9c20-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:03:58.310554 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:58.310558 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d999e2e2-910d-4c23-b784-31b2836a9c20-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:03:58.310769 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:58.310567 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d999e2e2-910d-4c23-b784-31b2836a9c20-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:03:58.523031 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:58.522998 5076 generic.go:356] "Generic (PLEG): container finished" podID="d999e2e2-910d-4c23-b784-31b2836a9c20" containerID="9e7f8afaeb949738a595a1eaa9d1571f0d66e2f34a06ac29d53a6053a3d88801" exitCode=0 Nov 02 23:03:58.523208 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:58.523039 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2" Nov 02 23:03:58.523208 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:58.523083 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2" event={"ID":"d999e2e2-910d-4c23-b784-31b2836a9c20","Type":"ContainerDied","Data":"9e7f8afaeb949738a595a1eaa9d1571f0d66e2f34a06ac29d53a6053a3d88801"} Nov 02 23:03:58.523208 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:58.523138 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2" event={"ID":"d999e2e2-910d-4c23-b784-31b2836a9c20","Type":"ContainerDied","Data":"5ad48be3c6909f4f9d14926538dc7dc678c391ed7b24884056ab295aa5bf4301"} Nov 02 23:03:58.523208 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:58.523169 5076 scope.go:122] "RemoveContainer" containerID="9e7f8afaeb949738a595a1eaa9d1571f0d66e2f34a06ac29d53a6053a3d88801" Nov 02 23:03:58.533365 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:58.533346 5076 scope.go:122] "RemoveContainer" containerID="9e7f8afaeb949738a595a1eaa9d1571f0d66e2f34a06ac29d53a6053a3d88801" Nov 02 23:03:58.533604 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:03:58.533586 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e7f8afaeb949738a595a1eaa9d1571f0d66e2f34a06ac29d53a6053a3d88801\": container with ID starting with 9e7f8afaeb949738a595a1eaa9d1571f0d66e2f34a06ac29d53a6053a3d88801 not found: ID does not exist" containerID="9e7f8afaeb949738a595a1eaa9d1571f0d66e2f34a06ac29d53a6053a3d88801" Nov 02 23:03:58.533652 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:58.533615 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e7f8afaeb949738a595a1eaa9d1571f0d66e2f34a06ac29d53a6053a3d88801"} err="failed to get container status \"9e7f8afaeb949738a595a1eaa9d1571f0d66e2f34a06ac29d53a6053a3d88801\": rpc error: code = NotFound desc = could not find container \"9e7f8afaeb949738a595a1eaa9d1571f0d66e2f34a06ac29d53a6053a3d88801\": container with ID starting with 9e7f8afaeb949738a595a1eaa9d1571f0d66e2f34a06ac29d53a6053a3d88801 not found: ID does not exist" Nov 02 23:03:58.678682 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:58.678651 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2"] Nov 02 23:03:58.682421 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:58.682390 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8a68c823-6c969cc4c4-vfgb2"] Nov 02 23:03:59.151083 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:03:59.149283 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d999e2e2-910d-4c23-b784-31b2836a9c20" path="/var/lib/kubelet/pods/d999e2e2-910d-4c23-b784-31b2836a9c20/volumes" Nov 02 23:04:00.206552 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:00.206515 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d0078993-57c5855b4c-htk5d"] Nov 02 23:04:00.305641 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:00.305606 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d0078993-6cd56965b7-5t6bm"] Nov 02 23:04:01.425230 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.425194 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n"] Nov 02 23:04:01.425701 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.425686 5076 cpu_manager.go:414] "containerMap: removing container" podUID="061b5c4a-b850-433e-ba62-3ae8de255048" containerName="payload-pre-processing" Nov 02 23:04:01.425757 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.425701 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="061b5c4a-b850-433e-ba62-3ae8de255048" containerName="payload-pre-processing" Nov 02 23:04:01.425757 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.425725 5076 cpu_manager.go:414] "containerMap: removing container" podUID="dfebc122-0b39-4c69-aa55-8d4aafa72905" containerName="payload-processing" Nov 02 23:04:01.425757 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.425730 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfebc122-0b39-4c69-aa55-8d4aafa72905" containerName="payload-processing" Nov 02 23:04:01.425757 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.425737 5076 cpu_manager.go:414] "containerMap: removing container" podUID="d999e2e2-910d-4c23-b784-31b2836a9c20" containerName="maas-api" Nov 02 23:04:01.425757 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.425745 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d999e2e2-910d-4c23-b784-31b2836a9c20" containerName="maas-api" Nov 02 23:04:01.425981 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.425821 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfebc122-0b39-4c69-aa55-8d4aafa72905" containerName="payload-processing" Nov 02 23:04:01.425981 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.425829 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="d999e2e2-910d-4c23-b784-31b2836a9c20" containerName="maas-api" Nov 02 23:04:01.425981 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.425839 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="061b5c4a-b850-433e-ba62-3ae8de255048" containerName="payload-pre-processing" Nov 02 23:04:01.430346 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.430325 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" Nov 02 23:04:01.501705 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.501674 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n"] Nov 02 23:04:01.538474 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.538444 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-14b321-openshift-default-dockercfg-ssm6b\"" Nov 02 23:04:01.544593 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.544565 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-istiod-ca-cert\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n\" (UID: \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" Nov 02 23:04:01.544727 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.544599 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cckfr\" (UniqueName: \"kubernetes.io/projected/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-kube-api-access-cckfr\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n\" (UID: \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" Nov 02 23:04:01.544727 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.544634 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-istio-podinfo\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n\" (UID: \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" Nov 02 23:04:01.544804 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.544760 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-istio-envoy\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n\" (UID: \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" Nov 02 23:04:01.544848 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.544826 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-workload-certs\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n\" (UID: \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" Nov 02 23:04:01.544940 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.544919 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-credential-socket\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n\" (UID: \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" Nov 02 23:04:01.544988 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.544968 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-istio-data\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n\" (UID: \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" Nov 02 23:04:01.545043 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.545027 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-istio-token\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n\" (UID: \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" Nov 02 23:04:01.545083 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.545063 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-workload-socket\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n\" (UID: \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" Nov 02 23:04:01.647499 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.647448 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-workload-certs\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n\" (UID: \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" Nov 02 23:04:01.647499 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.647507 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-credential-socket\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n\" (UID: \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" Nov 02 23:04:01.647722 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.647538 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-istio-data\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n\" (UID: \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" Nov 02 23:04:01.647722 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.647567 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-istio-token\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n\" (UID: \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" Nov 02 23:04:01.647722 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.647607 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-workload-socket\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n\" (UID: \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" Nov 02 23:04:01.647722 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.647633 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-istiod-ca-cert\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n\" (UID: \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" Nov 02 23:04:01.647722 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.647649 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cckfr\" (UniqueName: \"kubernetes.io/projected/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-kube-api-access-cckfr\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n\" (UID: \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" Nov 02 23:04:01.647722 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.647679 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-istio-podinfo\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n\" (UID: \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" Nov 02 23:04:01.647722 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.647718 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-istio-envoy\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n\" (UID: \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" Nov 02 23:04:01.648625 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.648596 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-workload-certs\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n\" (UID: \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" Nov 02 23:04:01.648867 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.648848 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-credential-socket\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n\" (UID: \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" Nov 02 23:04:01.648944 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.648854 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-istiod-ca-cert\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n\" (UID: \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" Nov 02 23:04:01.649137 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.649113 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-workload-socket\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n\" (UID: \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" Nov 02 23:04:01.649199 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.649161 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-istio-data\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n\" (UID: \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" Nov 02 23:04:01.651265 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.650430 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-istio-envoy\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n\" (UID: \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" Nov 02 23:04:01.656018 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.652055 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-istio-podinfo\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n\" (UID: \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" Nov 02 23:04:01.702280 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.702246 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-istio-token\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n\" (UID: \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" Nov 02 23:04:01.702447 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.702317 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cckfr\" (UniqueName: \"kubernetes.io/projected/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-kube-api-access-cckfr\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n\" (UID: \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" Nov 02 23:04:01.746353 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:01.746317 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" Nov 02 23:04:02.089758 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:02.088096 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bdaf93e3-nhjqv"] Nov 02 23:04:02.094941 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:02.094910 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bdaf93e3-nhjqv" Nov 02 23:04:02.142685 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:02.142649 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bdaf93e3-nhjqv"] Nov 02 23:04:02.144836 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:02.144809 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:04:02.144979 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:02.144874 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:04:02.152337 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:02.152310 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s56gt\" (UniqueName: \"kubernetes.io/projected/fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f-kube-api-access-s56gt\") pod \"maas-api-revoke-keys-e2e-derive-bdaf93e3-nhjqv\" (UID: \"fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bdaf93e3-nhjqv" Nov 02 23:04:02.152460 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:02.152351 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-bdaf93e3-nhjqv\" (UID: \"fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bdaf93e3-nhjqv" Nov 02 23:04:02.152530 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:02.152499 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-bdaf93e3-nhjqv\" (UID: \"fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bdaf93e3-nhjqv" Nov 02 23:04:02.209134 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:02.209111 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n"] Nov 02 23:04:02.211317 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:04:02.211286 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6b0b2ba_6e13_4d79_b2b1_243f80b5703c.slice/crio-e18c50700f0191f3a1140f770e2479e16fc6a40a85e70681bbd4e4424ba6330a WatchSource:0}: Error finding container e18c50700f0191f3a1140f770e2479e16fc6a40a85e70681bbd4e4424ba6330a: Status 404 returned error can't find the container with id e18c50700f0191f3a1140f770e2479e16fc6a40a85e70681bbd4e4424ba6330a Nov 02 23:04:02.213520 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:02.213489 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:04:02.213598 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:02.213547 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:04:02.213598 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:02.213576 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:04:02.253905 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:02.253864 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-bdaf93e3-nhjqv\" (UID: \"fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bdaf93e3-nhjqv" Nov 02 23:04:02.254056 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:02.254039 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-bdaf93e3-nhjqv\" (UID: \"fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bdaf93e3-nhjqv" Nov 02 23:04:02.254111 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:02.254097 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s56gt\" (UniqueName: \"kubernetes.io/projected/fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f-kube-api-access-s56gt\") pod \"maas-api-revoke-keys-e2e-derive-bdaf93e3-nhjqv\" (UID: \"fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bdaf93e3-nhjqv" Nov 02 23:04:02.254621 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:02.254604 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-bdaf93e3-nhjqv\" (UID: \"fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bdaf93e3-nhjqv" Nov 02 23:04:02.256057 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:02.256041 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-bdaf93e3-nhjqv\" (UID: \"fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bdaf93e3-nhjqv" Nov 02 23:04:02.308099 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:02.308069 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s56gt\" (UniqueName: \"kubernetes.io/projected/fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f-kube-api-access-s56gt\") pod \"maas-api-revoke-keys-e2e-derive-bdaf93e3-nhjqv\" (UID: \"fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bdaf93e3-nhjqv" Nov 02 23:04:02.408547 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:02.408461 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bdaf93e3-nhjqv" Nov 02 23:04:02.548508 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:02.548473 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" event={"ID":"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c","Type":"ContainerStarted","Data":"e1f08f0a7fc6972d298961353624a3625a1ae5eeaea913dabd22615b1b391dd9"} Nov 02 23:04:02.549003 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:02.548526 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" event={"ID":"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c","Type":"ContainerStarted","Data":"e18c50700f0191f3a1140f770e2479e16fc6a40a85e70681bbd4e4424ba6330a"} Nov 02 23:04:02.661116 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:02.661012 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" podStartSLOduration=1.660998406 podStartE2EDuration="1.660998406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:04:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:04:02.660406014 +0000 UTC m=+691.628937798" watchObservedRunningTime="2026-11-02 23:04:02.660998406 +0000 UTC m=+691.629530191" Nov 02 23:04:02.746855 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:02.746821 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" Nov 02 23:04:02.786387 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:02.786350 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bdaf93e3-nhjqv"] Nov 02 23:04:03.554953 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:03.554920 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bdaf93e3-nhjqv" event={"ID":"fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f","Type":"ContainerStarted","Data":"33884e524e4eae82c239771395cd683277e0a0196228f7605a548f607ac7c28b"} Nov 02 23:04:03.753235 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:03.753201 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" Nov 02 23:04:04.560180 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:04.560145 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" Nov 02 23:04:04.561112 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:04.561088 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" Nov 02 23:04:05.207677 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:05.207634 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d0078993-57c5855b4c-htk5d" podUID="f359fa32-c147-4ddd-8b34-709bec74a203" containerName="payload-processing" containerID="cri-o://64a2ad11044456abb9c53fa3d6df9f9a47bb2436ea83edd0ac3654e96379793e" gracePeriod=25 Nov 02 23:04:05.306462 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:05.306421 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0078993-6cd56965b7-5t6bm" podUID="5de5725b-cf19-4e08-8a28-cc2be38ff76f" containerName="payload-pre-processing" containerID="cri-o://f960e5e88ce2a4f0f92530f6e3df10cb9ac49c4f74997e3f04f41f7342096a16" gracePeriod=25 Nov 02 23:04:05.493489 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:05.493463 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d0078993-57c5855b4c-htk5d" Nov 02 23:04:05.568691 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:05.568656 5076 generic.go:356] "Generic (PLEG): container finished" podID="f359fa32-c147-4ddd-8b34-709bec74a203" containerID="64a2ad11044456abb9c53fa3d6df9f9a47bb2436ea83edd0ac3654e96379793e" exitCode=0 Nov 02 23:04:05.569364 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:05.568961 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d0078993-57c5855b4c-htk5d" Nov 02 23:04:05.569364 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:05.569144 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d0078993-57c5855b4c-htk5d" event={"ID":"f359fa32-c147-4ddd-8b34-709bec74a203","Type":"ContainerDied","Data":"64a2ad11044456abb9c53fa3d6df9f9a47bb2436ea83edd0ac3654e96379793e"} Nov 02 23:04:05.569364 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:05.569189 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d0078993-57c5855b4c-htk5d" event={"ID":"f359fa32-c147-4ddd-8b34-709bec74a203","Type":"ContainerDied","Data":"e732651cf10816a5e761effdfb33ac7a1dba1cb158b6e31e0f54dc2448dc6197"} Nov 02 23:04:05.569364 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:05.569220 5076 scope.go:122] "RemoveContainer" containerID="64a2ad11044456abb9c53fa3d6df9f9a47bb2436ea83edd0ac3654e96379793e" Nov 02 23:04:05.572269 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:05.572136 5076 generic.go:356] "Generic (PLEG): container finished" podID="5de5725b-cf19-4e08-8a28-cc2be38ff76f" containerID="f960e5e88ce2a4f0f92530f6e3df10cb9ac49c4f74997e3f04f41f7342096a16" exitCode=0 Nov 02 23:04:05.572269 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:05.572189 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0078993-6cd56965b7-5t6bm" event={"ID":"5de5725b-cf19-4e08-8a28-cc2be38ff76f","Type":"ContainerDied","Data":"f960e5e88ce2a4f0f92530f6e3df10cb9ac49c4f74997e3f04f41f7342096a16"} Nov 02 23:04:05.593970 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:05.593948 5076 scope.go:122] "RemoveContainer" containerID="64a2ad11044456abb9c53fa3d6df9f9a47bb2436ea83edd0ac3654e96379793e" Nov 02 23:04:05.594409 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:04:05.594385 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64a2ad11044456abb9c53fa3d6df9f9a47bb2436ea83edd0ac3654e96379793e\": container with ID starting with 64a2ad11044456abb9c53fa3d6df9f9a47bb2436ea83edd0ac3654e96379793e not found: ID does not exist" containerID="64a2ad11044456abb9c53fa3d6df9f9a47bb2436ea83edd0ac3654e96379793e" Nov 02 23:04:05.594504 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:05.594414 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64a2ad11044456abb9c53fa3d6df9f9a47bb2436ea83edd0ac3654e96379793e"} err="failed to get container status \"64a2ad11044456abb9c53fa3d6df9f9a47bb2436ea83edd0ac3654e96379793e\": rpc error: code = NotFound desc = could not find container \"64a2ad11044456abb9c53fa3d6df9f9a47bb2436ea83edd0ac3654e96379793e\": container with ID starting with 64a2ad11044456abb9c53fa3d6df9f9a47bb2436ea83edd0ac3654e96379793e not found: ID does not exist" Nov 02 23:04:05.598312 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:05.598292 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0078993-6cd56965b7-5t6bm" Nov 02 23:04:05.602043 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:05.602022 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f359fa32-c147-4ddd-8b34-709bec74a203-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f359fa32-c147-4ddd-8b34-709bec74a203-plugins-config-volume\") pod \"f359fa32-c147-4ddd-8b34-709bec74a203\" (UID: \"f359fa32-c147-4ddd-8b34-709bec74a203\") " Nov 02 23:04:05.602156 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:05.602081 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f359fa32-c147-4ddd-8b34-709bec74a203-kube-api-access-rpf55\" (UniqueName: \"kubernetes.io/projected/f359fa32-c147-4ddd-8b34-709bec74a203-kube-api-access-rpf55\") pod \"f359fa32-c147-4ddd-8b34-709bec74a203\" (UID: \"f359fa32-c147-4ddd-8b34-709bec74a203\") " Nov 02 23:04:05.602470 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:05.602441 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f359fa32-c147-4ddd-8b34-709bec74a203-plugins-config-volume" pod "f359fa32-c147-4ddd-8b34-709bec74a203" (UID: "f359fa32-c147-4ddd-8b34-709bec74a203"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:04:05.603347 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:05.603091 5076 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f359fa32-c147-4ddd-8b34-709bec74a203-plugins-config-volume\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:04:05.606513 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:05.606485 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f359fa32-c147-4ddd-8b34-709bec74a203-kube-api-access-rpf55" pod "f359fa32-c147-4ddd-8b34-709bec74a203" (UID: "f359fa32-c147-4ddd-8b34-709bec74a203"). InnerVolumeSpecName "kube-api-access-rpf55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:04:05.704425 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:05.704393 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5de5725b-cf19-4e08-8a28-cc2be38ff76f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5de5725b-cf19-4e08-8a28-cc2be38ff76f-plugins-config-volume\") pod \"5de5725b-cf19-4e08-8a28-cc2be38ff76f\" (UID: \"5de5725b-cf19-4e08-8a28-cc2be38ff76f\") " Nov 02 23:04:05.704610 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:05.704525 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5de5725b-cf19-4e08-8a28-cc2be38ff76f-kube-api-access-mzxzq\" (UniqueName: \"kubernetes.io/projected/5de5725b-cf19-4e08-8a28-cc2be38ff76f-kube-api-access-mzxzq\") pod \"5de5725b-cf19-4e08-8a28-cc2be38ff76f\" (UID: \"5de5725b-cf19-4e08-8a28-cc2be38ff76f\") " Nov 02 23:04:05.704832 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:05.704799 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5de5725b-cf19-4e08-8a28-cc2be38ff76f-plugins-config-volume" pod "5de5725b-cf19-4e08-8a28-cc2be38ff76f" (UID: "5de5725b-cf19-4e08-8a28-cc2be38ff76f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:04:05.705035 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:05.705015 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpf55\" (UniqueName: \"kubernetes.io/projected/f359fa32-c147-4ddd-8b34-709bec74a203-kube-api-access-rpf55\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:04:05.705101 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:05.705041 5076 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5de5725b-cf19-4e08-8a28-cc2be38ff76f-plugins-config-volume\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:04:05.707431 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:05.707407 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5de5725b-cf19-4e08-8a28-cc2be38ff76f-kube-api-access-mzxzq" pod "5de5725b-cf19-4e08-8a28-cc2be38ff76f" (UID: "5de5725b-cf19-4e08-8a28-cc2be38ff76f"). InnerVolumeSpecName "kube-api-access-mzxzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:04:05.806706 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:05.806629 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzxzq\" (UniqueName: \"kubernetes.io/projected/5de5725b-cf19-4e08-8a28-cc2be38ff76f-kube-api-access-mzxzq\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:04:05.988620 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:05.988571 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d0078993-57c5855b4c-htk5d"] Nov 02 23:04:05.992255 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:05.992229 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d0078993-57c5855b4c-htk5d"] Nov 02 23:04:06.580062 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:06.579984 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0078993-6cd56965b7-5t6bm" event={"ID":"5de5725b-cf19-4e08-8a28-cc2be38ff76f","Type":"ContainerDied","Data":"3d77e9ae697bfc78af8a7d9ca2b252f7568a331a2ebf58fd4200dc0d78820ece"} Nov 02 23:04:06.580062 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:06.580010 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0078993-6cd56965b7-5t6bm" Nov 02 23:04:06.580523 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:06.580094 5076 scope.go:122] "RemoveContainer" containerID="f960e5e88ce2a4f0f92530f6e3df10cb9ac49c4f74997e3f04f41f7342096a16" Nov 02 23:04:06.740362 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:06.740328 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d0078993-6cd56965b7-5t6bm"] Nov 02 23:04:06.748292 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:06.748257 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d0078993-6cd56965b7-5t6bm"] Nov 02 23:04:07.148510 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:07.148478 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5de5725b-cf19-4e08-8a28-cc2be38ff76f" path="/var/lib/kubelet/pods/5de5725b-cf19-4e08-8a28-cc2be38ff76f/volumes" Nov 02 23:04:07.148856 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:07.148843 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f359fa32-c147-4ddd-8b34-709bec74a203" path="/var/lib/kubelet/pods/f359fa32-c147-4ddd-8b34-709bec74a203/volumes" Nov 02 23:04:07.590405 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:07.590373 5076 generic.go:356] "Generic (PLEG): container finished" podID="fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f" containerID="2f7c0ca7d96e9c02d1ec1fc285ac921b8f62e66456638404450a58729df3185e" exitCode=0 Nov 02 23:04:07.590835 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:07.590436 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bdaf93e3-nhjqv" event={"ID":"fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f","Type":"ContainerDied","Data":"2f7c0ca7d96e9c02d1ec1fc285ac921b8f62e66456638404450a58729df3185e"} Nov 02 23:04:07.842482 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:07.842407 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq"] Nov 02 23:04:07.842939 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:07.842924 5076 cpu_manager.go:414] "containerMap: removing container" podUID="5de5725b-cf19-4e08-8a28-cc2be38ff76f" containerName="payload-pre-processing" Nov 02 23:04:07.842993 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:07.842942 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5de5725b-cf19-4e08-8a28-cc2be38ff76f" containerName="payload-pre-processing" Nov 02 23:04:07.842993 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:07.842964 5076 cpu_manager.go:414] "containerMap: removing container" podUID="f359fa32-c147-4ddd-8b34-709bec74a203" containerName="payload-processing" Nov 02 23:04:07.842993 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:07.842969 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f359fa32-c147-4ddd-8b34-709bec74a203" containerName="payload-processing" Nov 02 23:04:07.843106 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:07.843094 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="f359fa32-c147-4ddd-8b34-709bec74a203" containerName="payload-processing" Nov 02 23:04:07.843149 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:07.843109 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="5de5725b-cf19-4e08-8a28-cc2be38ff76f" containerName="payload-pre-processing" Nov 02 23:04:07.846474 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:07.846454 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" Nov 02 23:04:07.936508 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:07.936469 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc003675-8adf-4c2d-af38-9e3ad5af69f4-istio-envoy\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq\" (UID: \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" Nov 02 23:04:07.936692 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:07.936515 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc003675-8adf-4c2d-af38-9e3ad5af69f4-istio-podinfo\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq\" (UID: \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" Nov 02 23:04:07.936692 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:07.936542 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svzks\" (UniqueName: \"kubernetes.io/projected/cc003675-8adf-4c2d-af38-9e3ad5af69f4-kube-api-access-svzks\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq\" (UID: \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" Nov 02 23:04:07.936692 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:07.936597 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc003675-8adf-4c2d-af38-9e3ad5af69f4-istio-token\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq\" (UID: \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" Nov 02 23:04:07.936692 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:07.936653 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc003675-8adf-4c2d-af38-9e3ad5af69f4-workload-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq\" (UID: \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" Nov 02 23:04:07.936947 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:07.936702 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc003675-8adf-4c2d-af38-9e3ad5af69f4-workload-certs\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq\" (UID: \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" Nov 02 23:04:07.936947 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:07.936725 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc003675-8adf-4c2d-af38-9e3ad5af69f4-istio-data\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq\" (UID: \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" Nov 02 23:04:07.936947 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:07.936800 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc003675-8adf-4c2d-af38-9e3ad5af69f4-istiod-ca-cert\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq\" (UID: \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" Nov 02 23:04:07.936947 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:07.936837 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc003675-8adf-4c2d-af38-9e3ad5af69f4-credential-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq\" (UID: \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" Nov 02 23:04:07.940769 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:07.939878 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq"] Nov 02 23:04:08.014109 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:07.996266 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-59f97a-openshift-default-dockercfg-r7nkq\"" Nov 02 23:04:08.038533 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:08.038494 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc003675-8adf-4c2d-af38-9e3ad5af69f4-istiod-ca-cert\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq\" (UID: \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" Nov 02 23:04:08.038738 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:08.038695 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc003675-8adf-4c2d-af38-9e3ad5af69f4-credential-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq\" (UID: \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" Nov 02 23:04:08.038819 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:08.038747 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc003675-8adf-4c2d-af38-9e3ad5af69f4-istio-envoy\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq\" (UID: \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" Nov 02 23:04:08.038819 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:08.038772 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc003675-8adf-4c2d-af38-9e3ad5af69f4-istio-podinfo\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq\" (UID: \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" Nov 02 23:04:08.038819 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:08.038802 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svzks\" (UniqueName: \"kubernetes.io/projected/cc003675-8adf-4c2d-af38-9e3ad5af69f4-kube-api-access-svzks\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq\" (UID: \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" Nov 02 23:04:08.039022 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:08.038871 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc003675-8adf-4c2d-af38-9e3ad5af69f4-istio-token\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq\" (UID: \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" Nov 02 23:04:08.039022 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:08.038961 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc003675-8adf-4c2d-af38-9e3ad5af69f4-workload-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq\" (UID: \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" Nov 02 23:04:08.039183 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:08.039158 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc003675-8adf-4c2d-af38-9e3ad5af69f4-credential-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq\" (UID: \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" Nov 02 23:04:08.039255 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:08.039219 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc003675-8adf-4c2d-af38-9e3ad5af69f4-workload-certs\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq\" (UID: \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" Nov 02 23:04:08.039319 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:08.039253 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc003675-8adf-4c2d-af38-9e3ad5af69f4-istio-data\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq\" (UID: \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" Nov 02 23:04:08.039376 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:08.039327 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc003675-8adf-4c2d-af38-9e3ad5af69f4-istiod-ca-cert\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq\" (UID: \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" Nov 02 23:04:08.039376 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:08.039342 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc003675-8adf-4c2d-af38-9e3ad5af69f4-workload-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq\" (UID: \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" Nov 02 23:04:08.039576 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:08.039555 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc003675-8adf-4c2d-af38-9e3ad5af69f4-workload-certs\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq\" (UID: \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" Nov 02 23:04:08.039661 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:08.039609 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc003675-8adf-4c2d-af38-9e3ad5af69f4-istio-data\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq\" (UID: \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" Nov 02 23:04:08.041246 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:08.041223 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc003675-8adf-4c2d-af38-9e3ad5af69f4-istio-envoy\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq\" (UID: \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" Nov 02 23:04:08.041330 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:08.041296 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc003675-8adf-4c2d-af38-9e3ad5af69f4-istio-podinfo\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq\" (UID: \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" Nov 02 23:04:08.093865 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:08.093792 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc003675-8adf-4c2d-af38-9e3ad5af69f4-istio-token\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq\" (UID: \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" Nov 02 23:04:08.094052 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:08.093957 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svzks\" (UniqueName: \"kubernetes.io/projected/cc003675-8adf-4c2d-af38-9e3ad5af69f4-kube-api-access-svzks\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq\" (UID: \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" Nov 02 23:04:08.159914 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:08.159849 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" Nov 02 23:04:08.396590 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:08.396503 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-16aef2f3-7649b9bf5f-8ngg9"] Nov 02 23:04:08.497447 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:08.497417 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-16aef2f3-794798c447-zpcwj"] Nov 02 23:04:08.578168 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:08.578142 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq"] Nov 02 23:04:08.579434 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:04:08.579405 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc003675_8adf_4c2d_af38_9e3ad5af69f4.slice/crio-eeda88117570750d051fb6cf5b1314a486a2bc1dc1d502d67a80620112e3e193 WatchSource:0}: Error finding container eeda88117570750d051fb6cf5b1314a486a2bc1dc1d502d67a80620112e3e193: Status 404 returned error can't find the container with id eeda88117570750d051fb6cf5b1314a486a2bc1dc1d502d67a80620112e3e193 Nov 02 23:04:08.581905 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:08.581861 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:04:08.581997 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:08.581955 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:04:08.582055 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:08.582001 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:04:08.596983 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:08.596958 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" event={"ID":"cc003675-8adf-4c2d-af38-9e3ad5af69f4","Type":"ContainerStarted","Data":"eeda88117570750d051fb6cf5b1314a486a2bc1dc1d502d67a80620112e3e193"} Nov 02 23:04:08.769794 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:08.769765 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bdaf93e3-nhjqv" Nov 02 23:04:08.846123 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:08.846092 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f-kube-api-access-s56gt\" (UniqueName: \"kubernetes.io/projected/fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f-kube-api-access-s56gt\") pod \"fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f\" (UID: \"fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f\") " Nov 02 23:04:08.846306 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:08.846270 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f-maas-api-xks-ca\") pod \"fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f\" (UID: \"fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f\") " Nov 02 23:04:08.846374 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:08.846334 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f-maas-api-openshift-ca\") pod \"fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f\" (UID: \"fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f\") " Nov 02 23:04:08.846835 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:08.846786 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f-maas-api-openshift-ca" pod "fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f" (UID: "fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:04:08.848754 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:08.848730 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f-maas-api-xks-ca" pod "fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f" (UID: "fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:04:08.848850 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:08.848827 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f-kube-api-access-s56gt" pod "fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f" (UID: "fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f"). InnerVolumeSpecName "kube-api-access-s56gt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:04:08.948138 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:08.948107 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f-maas-api-xks-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:04:08.948138 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:08.948136 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f-maas-api-openshift-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:04:08.948138 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:08.948150 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s56gt\" (UniqueName: \"kubernetes.io/projected/fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f-kube-api-access-s56gt\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:04:09.603188 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:09.603155 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bdaf93e3-nhjqv" Nov 02 23:04:09.603585 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:09.603158 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bdaf93e3-nhjqv" event={"ID":"fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f","Type":"ContainerDied","Data":"33884e524e4eae82c239771395cd683277e0a0196228f7605a548f607ac7c28b"} Nov 02 23:04:09.603585 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:09.603299 5076 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33884e524e4eae82c239771395cd683277e0a0196228f7605a548f607ac7c28b" Nov 02 23:04:09.604995 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:09.604968 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" event={"ID":"cc003675-8adf-4c2d-af38-9e3ad5af69f4","Type":"ContainerStarted","Data":"8330bd727513640206cf1fe03726eba7c69976ae3a5bdfec0f9e67bd2ce5254d"} Nov 02 23:04:09.716304 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:09.716249 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" podStartSLOduration=2.716233564 podStartE2EDuration="2.716233564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:04:09.715025737 +0000 UTC m=+698.683557522" watchObservedRunningTime="2026-11-02 23:04:09.716233564 +0000 UTC m=+698.684765410" Nov 02 23:04:09.899040 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:04:09.898933 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:04:10.161125 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:10.161047 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" Nov 02 23:04:10.166284 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:10.166258 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" Nov 02 23:04:10.610700 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:10.610670 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" Nov 02 23:04:10.611575 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:10.611557 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" Nov 02 23:04:13.114927 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:13.114880 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bdaf93e3-7446956765-qf8nk"] Nov 02 23:04:13.215591 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:13.215554 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bdaf93e3-57cf9555-cc2rb"] Nov 02 23:04:13.397912 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:13.397802 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-16aef2f3-7649b9bf5f-8ngg9" podUID="27c66990-66df-4a66-b3ec-e66a0e8fd540" containerName="payload-processing" containerID="cri-o://fee83b119819b17f9280496dcc1f7b9f8d7c615bfb9d58d1b8cc12225d61b2e4" gracePeriod=25 Nov 02 23:04:13.498063 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:13.498020 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-16aef2f3-794798c447-zpcwj" podUID="d0f6b370-0c0e-45a5-b612-c8a1b3036dc2" containerName="payload-pre-processing" containerID="cri-o://2521652d19181a45dfbc2793d66400326bc4d12d70930730d2b3f7e3f5d6d671" gracePeriod=25 Nov 02 23:04:13.629681 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:13.629646 5076 generic.go:356] "Generic (PLEG): container finished" podID="27c66990-66df-4a66-b3ec-e66a0e8fd540" containerID="fee83b119819b17f9280496dcc1f7b9f8d7c615bfb9d58d1b8cc12225d61b2e4" exitCode=0 Nov 02 23:04:13.629849 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:13.629724 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-16aef2f3-7649b9bf5f-8ngg9" event={"ID":"27c66990-66df-4a66-b3ec-e66a0e8fd540","Type":"ContainerDied","Data":"fee83b119819b17f9280496dcc1f7b9f8d7c615bfb9d58d1b8cc12225d61b2e4"} Nov 02 23:04:13.631733 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:13.631704 5076 generic.go:356] "Generic (PLEG): container finished" podID="d0f6b370-0c0e-45a5-b612-c8a1b3036dc2" containerID="2521652d19181a45dfbc2793d66400326bc4d12d70930730d2b3f7e3f5d6d671" exitCode=0 Nov 02 23:04:13.631879 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:13.631763 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-16aef2f3-794798c447-zpcwj" event={"ID":"d0f6b370-0c0e-45a5-b612-c8a1b3036dc2","Type":"ContainerDied","Data":"2521652d19181a45dfbc2793d66400326bc4d12d70930730d2b3f7e3f5d6d671"} Nov 02 23:04:13.719238 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:13.719213 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-16aef2f3-7649b9bf5f-8ngg9" Nov 02 23:04:13.783382 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:13.783359 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-16aef2f3-794798c447-zpcwj" Nov 02 23:04:13.802544 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:13.802507 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27c66990-66df-4a66-b3ec-e66a0e8fd540-kube-api-access-bmj4b\" (UniqueName: \"kubernetes.io/projected/27c66990-66df-4a66-b3ec-e66a0e8fd540-kube-api-access-bmj4b\") pod \"27c66990-66df-4a66-b3ec-e66a0e8fd540\" (UID: \"27c66990-66df-4a66-b3ec-e66a0e8fd540\") " Nov 02 23:04:13.802719 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:13.802576 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27c66990-66df-4a66-b3ec-e66a0e8fd540-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c66990-66df-4a66-b3ec-e66a0e8fd540-plugins-config-volume\") pod \"27c66990-66df-4a66-b3ec-e66a0e8fd540\" (UID: \"27c66990-66df-4a66-b3ec-e66a0e8fd540\") " Nov 02 23:04:13.803012 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:13.802989 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27c66990-66df-4a66-b3ec-e66a0e8fd540-plugins-config-volume" pod "27c66990-66df-4a66-b3ec-e66a0e8fd540" (UID: "27c66990-66df-4a66-b3ec-e66a0e8fd540"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:04:13.804634 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:13.804611 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27c66990-66df-4a66-b3ec-e66a0e8fd540-kube-api-access-bmj4b" pod "27c66990-66df-4a66-b3ec-e66a0e8fd540" (UID: "27c66990-66df-4a66-b3ec-e66a0e8fd540"). InnerVolumeSpecName "kube-api-access-bmj4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:04:13.903507 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:13.903477 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0f6b370-0c0e-45a5-b612-c8a1b3036dc2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0f6b370-0c0e-45a5-b612-c8a1b3036dc2-plugins-config-volume\") pod \"d0f6b370-0c0e-45a5-b612-c8a1b3036dc2\" (UID: \"d0f6b370-0c0e-45a5-b612-c8a1b3036dc2\") " Nov 02 23:04:13.903693 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:13.903547 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0f6b370-0c0e-45a5-b612-c8a1b3036dc2-kube-api-access-xknvg\" (UniqueName: \"kubernetes.io/projected/d0f6b370-0c0e-45a5-b612-c8a1b3036dc2-kube-api-access-xknvg\") pod \"d0f6b370-0c0e-45a5-b612-c8a1b3036dc2\" (UID: \"d0f6b370-0c0e-45a5-b612-c8a1b3036dc2\") " Nov 02 23:04:13.903935 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:13.903878 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0f6b370-0c0e-45a5-b612-c8a1b3036dc2-plugins-config-volume" pod "d0f6b370-0c0e-45a5-b612-c8a1b3036dc2" (UID: "d0f6b370-0c0e-45a5-b612-c8a1b3036dc2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:04:13.904039 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:13.904025 5076 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0f6b370-0c0e-45a5-b612-c8a1b3036dc2-plugins-config-volume\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:04:13.904095 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:13.904043 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmj4b\" (UniqueName: \"kubernetes.io/projected/27c66990-66df-4a66-b3ec-e66a0e8fd540-kube-api-access-bmj4b\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:04:13.904095 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:13.904053 5076 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c66990-66df-4a66-b3ec-e66a0e8fd540-plugins-config-volume\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:04:13.905664 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:13.905645 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0f6b370-0c0e-45a5-b612-c8a1b3036dc2-kube-api-access-xknvg" pod "d0f6b370-0c0e-45a5-b612-c8a1b3036dc2" (UID: "d0f6b370-0c0e-45a5-b612-c8a1b3036dc2"). InnerVolumeSpecName "kube-api-access-xknvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:04:14.005566 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:14.005436 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xknvg\" (UniqueName: \"kubernetes.io/projected/d0f6b370-0c0e-45a5-b612-c8a1b3036dc2-kube-api-access-xknvg\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:04:14.638475 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:14.638442 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-16aef2f3-7649b9bf5f-8ngg9" Nov 02 23:04:14.638954 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:14.638476 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-16aef2f3-7649b9bf5f-8ngg9" event={"ID":"27c66990-66df-4a66-b3ec-e66a0e8fd540","Type":"ContainerDied","Data":"e6d1d7c6c086d929c38206be500c3b848c00943b35db1d3fca43f348c7e72f5f"} Nov 02 23:04:14.638954 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:14.638547 5076 scope.go:122] "RemoveContainer" containerID="fee83b119819b17f9280496dcc1f7b9f8d7c615bfb9d58d1b8cc12225d61b2e4" Nov 02 23:04:14.640265 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:14.640250 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-16aef2f3-794798c447-zpcwj" Nov 02 23:04:14.640265 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:14.640248 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-16aef2f3-794798c447-zpcwj" event={"ID":"d0f6b370-0c0e-45a5-b612-c8a1b3036dc2","Type":"ContainerDied","Data":"a24d673f0a1f5af8b86e447275c461b045c5ee5b5c3e01dc77c77a2221604f7d"} Nov 02 23:04:14.650227 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:14.650209 5076 scope.go:122] "RemoveContainer" containerID="2521652d19181a45dfbc2793d66400326bc4d12d70930730d2b3f7e3f5d6d671" Nov 02 23:04:14.793884 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:14.793849 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-16aef2f3-7649b9bf5f-8ngg9"] Nov 02 23:04:14.798704 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:14.798675 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-16aef2f3-7649b9bf5f-8ngg9"] Nov 02 23:04:14.902818 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:14.902783 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-16aef2f3-794798c447-zpcwj"] Nov 02 23:04:14.906083 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:14.906056 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-16aef2f3-794798c447-zpcwj"] Nov 02 23:04:15.147028 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:15.146951 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27c66990-66df-4a66-b3ec-e66a0e8fd540" path="/var/lib/kubelet/pods/27c66990-66df-4a66-b3ec-e66a0e8fd540/volumes" Nov 02 23:04:15.147336 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:15.147323 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0f6b370-0c0e-45a5-b612-c8a1b3036dc2" path="/var/lib/kubelet/pods/d0f6b370-0c0e-45a5-b612-c8a1b3036dc2/volumes" Nov 02 23:04:18.116195 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:18.116142 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-bdaf93e3-7446956765-qf8nk" podUID="5769d8f9-955f-43b4-9f73-433e625aea56" containerName="payload-processing" containerID="cri-o://8ba0019109705eefcc2d98025c23f8fd3e7cf8adfab6082241494e3c17e45201" gracePeriod=25 Nov 02 23:04:18.216387 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:18.216339 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-bdaf93e3-57cf9555-cc2rb" podUID="6b1d58ec-753e-46e8-882d-b0c3fe6f91cc" containerName="payload-pre-processing" containerID="cri-o://03a03c27394468acca6d7c358155304d6d8537017f7de32c647d215ea4c1ce36" gracePeriod=25 Nov 02 23:04:18.372284 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:18.372208 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-bdaf93e3-7446956765-qf8nk" Nov 02 23:04:18.452777 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:18.452748 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5769d8f9-955f-43b4-9f73-433e625aea56-kube-api-access-p8rb5\" (UniqueName: \"kubernetes.io/projected/5769d8f9-955f-43b4-9f73-433e625aea56-kube-api-access-p8rb5\") pod \"5769d8f9-955f-43b4-9f73-433e625aea56\" (UID: \"5769d8f9-955f-43b4-9f73-433e625aea56\") " Nov 02 23:04:18.452957 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:18.452794 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5769d8f9-955f-43b4-9f73-433e625aea56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5769d8f9-955f-43b4-9f73-433e625aea56-plugins-config-volume\") pod \"5769d8f9-955f-43b4-9f73-433e625aea56\" (UID: \"5769d8f9-955f-43b4-9f73-433e625aea56\") " Nov 02 23:04:18.453236 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:18.453208 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5769d8f9-955f-43b4-9f73-433e625aea56-plugins-config-volume" pod "5769d8f9-955f-43b4-9f73-433e625aea56" (UID: "5769d8f9-955f-43b4-9f73-433e625aea56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:04:18.454832 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:18.454807 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5769d8f9-955f-43b4-9f73-433e625aea56-kube-api-access-p8rb5" pod "5769d8f9-955f-43b4-9f73-433e625aea56" (UID: "5769d8f9-955f-43b4-9f73-433e625aea56"). InnerVolumeSpecName "kube-api-access-p8rb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:04:18.475669 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:18.475646 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bdaf93e3-57cf9555-cc2rb" Nov 02 23:04:18.554252 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:18.554215 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b1d58ec-753e-46e8-882d-b0c3fe6f91cc-kube-api-access-z4v29\" (UniqueName: \"kubernetes.io/projected/6b1d58ec-753e-46e8-882d-b0c3fe6f91cc-kube-api-access-z4v29\") pod \"6b1d58ec-753e-46e8-882d-b0c3fe6f91cc\" (UID: \"6b1d58ec-753e-46e8-882d-b0c3fe6f91cc\") " Nov 02 23:04:18.554446 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:18.554331 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b1d58ec-753e-46e8-882d-b0c3fe6f91cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b1d58ec-753e-46e8-882d-b0c3fe6f91cc-plugins-config-volume\") pod \"6b1d58ec-753e-46e8-882d-b0c3fe6f91cc\" (UID: \"6b1d58ec-753e-46e8-882d-b0c3fe6f91cc\") " Nov 02 23:04:18.554622 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:18.554606 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8rb5\" (UniqueName: \"kubernetes.io/projected/5769d8f9-955f-43b4-9f73-433e625aea56-kube-api-access-p8rb5\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:04:18.554673 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:18.554627 5076 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5769d8f9-955f-43b4-9f73-433e625aea56-plugins-config-volume\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:04:18.554716 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:18.554691 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b1d58ec-753e-46e8-882d-b0c3fe6f91cc-plugins-config-volume" pod "6b1d58ec-753e-46e8-882d-b0c3fe6f91cc" (UID: "6b1d58ec-753e-46e8-882d-b0c3fe6f91cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:04:18.556377 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:18.556357 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b1d58ec-753e-46e8-882d-b0c3fe6f91cc-kube-api-access-z4v29" pod "6b1d58ec-753e-46e8-882d-b0c3fe6f91cc" (UID: "6b1d58ec-753e-46e8-882d-b0c3fe6f91cc"). InnerVolumeSpecName "kube-api-access-z4v29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:04:18.656012 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:18.655930 5076 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b1d58ec-753e-46e8-882d-b0c3fe6f91cc-plugins-config-volume\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:04:18.656012 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:18.655961 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4v29\" (UniqueName: \"kubernetes.io/projected/6b1d58ec-753e-46e8-882d-b0c3fe6f91cc-kube-api-access-z4v29\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:04:18.663709 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:18.663681 5076 generic.go:356] "Generic (PLEG): container finished" podID="6b1d58ec-753e-46e8-882d-b0c3fe6f91cc" containerID="03a03c27394468acca6d7c358155304d6d8537017f7de32c647d215ea4c1ce36" exitCode=0 Nov 02 23:04:18.663842 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:18.663729 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bdaf93e3-57cf9555-cc2rb" Nov 02 23:04:18.663842 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:18.663767 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bdaf93e3-57cf9555-cc2rb" event={"ID":"6b1d58ec-753e-46e8-882d-b0c3fe6f91cc","Type":"ContainerDied","Data":"03a03c27394468acca6d7c358155304d6d8537017f7de32c647d215ea4c1ce36"} Nov 02 23:04:18.663961 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:18.663847 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bdaf93e3-57cf9555-cc2rb" event={"ID":"6b1d58ec-753e-46e8-882d-b0c3fe6f91cc","Type":"ContainerDied","Data":"119eb0af604bd14ce159ae058dc430b8400783d57e28176a5dda95c0b57fd49d"} Nov 02 23:04:18.663961 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:18.663876 5076 scope.go:122] "RemoveContainer" containerID="03a03c27394468acca6d7c358155304d6d8537017f7de32c647d215ea4c1ce36" Nov 02 23:04:18.665181 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:18.665157 5076 generic.go:356] "Generic (PLEG): container finished" podID="5769d8f9-955f-43b4-9f73-433e625aea56" containerID="8ba0019109705eefcc2d98025c23f8fd3e7cf8adfab6082241494e3c17e45201" exitCode=0 Nov 02 23:04:18.665268 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:18.665192 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-bdaf93e3-7446956765-qf8nk" event={"ID":"5769d8f9-955f-43b4-9f73-433e625aea56","Type":"ContainerDied","Data":"8ba0019109705eefcc2d98025c23f8fd3e7cf8adfab6082241494e3c17e45201"} Nov 02 23:04:18.665268 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:18.665207 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-bdaf93e3-7446956765-qf8nk" Nov 02 23:04:18.665268 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:18.665232 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-bdaf93e3-7446956765-qf8nk" event={"ID":"5769d8f9-955f-43b4-9f73-433e625aea56","Type":"ContainerDied","Data":"f054277b06d768de29149e9585f49bb17abf93b48ede7956036ebe1967227598"} Nov 02 23:04:18.674096 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:18.674078 5076 scope.go:122] "RemoveContainer" containerID="03a03c27394468acca6d7c358155304d6d8537017f7de32c647d215ea4c1ce36" Nov 02 23:04:18.674421 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:04:18.674401 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03a03c27394468acca6d7c358155304d6d8537017f7de32c647d215ea4c1ce36\": container with ID starting with 03a03c27394468acca6d7c358155304d6d8537017f7de32c647d215ea4c1ce36 not found: ID does not exist" containerID="03a03c27394468acca6d7c358155304d6d8537017f7de32c647d215ea4c1ce36" Nov 02 23:04:18.674493 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:18.674429 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03a03c27394468acca6d7c358155304d6d8537017f7de32c647d215ea4c1ce36"} err="failed to get container status \"03a03c27394468acca6d7c358155304d6d8537017f7de32c647d215ea4c1ce36\": rpc error: code = NotFound desc = could not find container \"03a03c27394468acca6d7c358155304d6d8537017f7de32c647d215ea4c1ce36\": container with ID starting with 03a03c27394468acca6d7c358155304d6d8537017f7de32c647d215ea4c1ce36 not found: ID does not exist" Nov 02 23:04:18.674493 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:18.674446 5076 scope.go:122] "RemoveContainer" containerID="8ba0019109705eefcc2d98025c23f8fd3e7cf8adfab6082241494e3c17e45201" Nov 02 23:04:18.683254 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:18.683235 5076 scope.go:122] "RemoveContainer" containerID="8ba0019109705eefcc2d98025c23f8fd3e7cf8adfab6082241494e3c17e45201" Nov 02 23:04:18.683508 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:04:18.683486 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ba0019109705eefcc2d98025c23f8fd3e7cf8adfab6082241494e3c17e45201\": container with ID starting with 8ba0019109705eefcc2d98025c23f8fd3e7cf8adfab6082241494e3c17e45201 not found: ID does not exist" containerID="8ba0019109705eefcc2d98025c23f8fd3e7cf8adfab6082241494e3c17e45201" Nov 02 23:04:18.683573 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:18.683515 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ba0019109705eefcc2d98025c23f8fd3e7cf8adfab6082241494e3c17e45201"} err="failed to get container status \"8ba0019109705eefcc2d98025c23f8fd3e7cf8adfab6082241494e3c17e45201\": rpc error: code = NotFound desc = could not find container \"8ba0019109705eefcc2d98025c23f8fd3e7cf8adfab6082241494e3c17e45201\": container with ID starting with 8ba0019109705eefcc2d98025c23f8fd3e7cf8adfab6082241494e3c17e45201 not found: ID does not exist" Nov 02 23:04:18.819918 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:18.819870 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bdaf93e3-7446956765-qf8nk"] Nov 02 23:04:18.822971 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:18.822945 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bdaf93e3-7446956765-qf8nk"] Nov 02 23:04:18.939999 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:18.939964 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bdaf93e3-57cf9555-cc2rb"] Nov 02 23:04:18.961596 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:18.961562 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bdaf93e3-57cf9555-cc2rb"] Nov 02 23:04:19.147159 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:19.147128 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5769d8f9-955f-43b4-9f73-433e625aea56" path="/var/lib/kubelet/pods/5769d8f9-955f-43b4-9f73-433e625aea56/volumes" Nov 02 23:04:19.147570 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:19.147556 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b1d58ec-753e-46e8-882d-b0c3fe6f91cc" path="/var/lib/kubelet/pods/6b1d58ec-753e-46e8-882d-b0c3fe6f91cc/volumes" Nov 02 23:04:20.588100 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:20.588065 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-59f97a-5c46c69c55-c68jw"] Nov 02 23:04:20.588573 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:20.588555 5076 cpu_manager.go:414] "containerMap: removing container" podUID="6b1d58ec-753e-46e8-882d-b0c3fe6f91cc" containerName="payload-pre-processing" Nov 02 23:04:20.588664 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:20.588575 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b1d58ec-753e-46e8-882d-b0c3fe6f91cc" containerName="payload-pre-processing" Nov 02 23:04:20.588664 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:20.588596 5076 cpu_manager.go:414] "containerMap: removing container" podUID="fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f" containerName="revoke-keys" Nov 02 23:04:20.588664 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:20.588602 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f" containerName="revoke-keys" Nov 02 23:04:20.588664 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:20.588617 5076 cpu_manager.go:414] "containerMap: removing container" podUID="27c66990-66df-4a66-b3ec-e66a0e8fd540" containerName="payload-processing" Nov 02 23:04:20.588664 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:20.588622 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27c66990-66df-4a66-b3ec-e66a0e8fd540" containerName="payload-processing" Nov 02 23:04:20.588664 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:20.588630 5076 cpu_manager.go:414] "containerMap: removing container" podUID="d0f6b370-0c0e-45a5-b612-c8a1b3036dc2" containerName="payload-pre-processing" Nov 02 23:04:20.588664 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:20.588634 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0f6b370-0c0e-45a5-b612-c8a1b3036dc2" containerName="payload-pre-processing" Nov 02 23:04:20.588664 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:20.588644 5076 cpu_manager.go:414] "containerMap: removing container" podUID="5769d8f9-955f-43b4-9f73-433e625aea56" containerName="payload-processing" Nov 02 23:04:20.588664 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:20.588648 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5769d8f9-955f-43b4-9f73-433e625aea56" containerName="payload-processing" Nov 02 23:04:20.588959 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:20.588729 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b1d58ec-753e-46e8-882d-b0c3fe6f91cc" containerName="payload-pre-processing" Nov 02 23:04:20.588959 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:20.588745 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f" containerName="revoke-keys" Nov 02 23:04:20.588959 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:20.588758 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="27c66990-66df-4a66-b3ec-e66a0e8fd540" containerName="payload-processing" Nov 02 23:04:20.588959 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:20.588764 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0f6b370-0c0e-45a5-b612-c8a1b3036dc2" containerName="payload-pre-processing" Nov 02 23:04:20.588959 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:20.588770 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="5769d8f9-955f-43b4-9f73-433e625aea56" containerName="payload-processing" Nov 02 23:04:20.593492 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:20.593472 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-59f97a-5c46c69c55-c68jw" Nov 02 23:04:20.660916 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:20.653721 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-59f97a-5c46c69c55-c68jw"] Nov 02 23:04:20.675200 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:20.675157 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xklgt\" (UniqueName: \"kubernetes.io/projected/b8ac8868-81dc-42dc-aed6-22e78b580eec-kube-api-access-xklgt\") pod \"payload-pre-processing-e2e-worker-w0-59f97a-5c46c69c55-c68jw\" (UID: \"b8ac8868-81dc-42dc-aed6-22e78b580eec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-59f97a-5c46c69c55-c68jw" Nov 02 23:04:20.675401 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:20.675289 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8ac8868-81dc-42dc-aed6-22e78b580eec-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-59f97a-5c46c69c55-c68jw\" (UID: \"b8ac8868-81dc-42dc-aed6-22e78b580eec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-59f97a-5c46c69c55-c68jw" Nov 02 23:04:20.698382 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:20.698348 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-59f97a\"" Nov 02 23:04:20.698382 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:20.698362 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-59f97a-dockercfg-hhfgh\"" Nov 02 23:04:20.743817 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:20.743783 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-59f97a-578d5df469-bmvsq"] Nov 02 23:04:20.747398 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:20.747380 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-59f97a-578d5df469-bmvsq" Nov 02 23:04:20.776314 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:20.776287 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xklgt\" (UniqueName: \"kubernetes.io/projected/b8ac8868-81dc-42dc-aed6-22e78b580eec-kube-api-access-xklgt\") pod \"payload-pre-processing-e2e-worker-w0-59f97a-5c46c69c55-c68jw\" (UID: \"b8ac8868-81dc-42dc-aed6-22e78b580eec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-59f97a-5c46c69c55-c68jw" Nov 02 23:04:20.776441 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:20.776368 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8ac8868-81dc-42dc-aed6-22e78b580eec-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-59f97a-5c46c69c55-c68jw\" (UID: \"b8ac8868-81dc-42dc-aed6-22e78b580eec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-59f97a-5c46c69c55-c68jw" Nov 02 23:04:20.777422 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:20.777398 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8ac8868-81dc-42dc-aed6-22e78b580eec-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-59f97a-5c46c69c55-c68jw\" (UID: \"b8ac8868-81dc-42dc-aed6-22e78b580eec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-59f97a-5c46c69c55-c68jw" Nov 02 23:04:20.800186 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:20.800152 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-59f97a-578d5df469-bmvsq"] Nov 02 23:04:20.830551 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:20.830511 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xklgt\" (UniqueName: \"kubernetes.io/projected/b8ac8868-81dc-42dc-aed6-22e78b580eec-kube-api-access-xklgt\") pod \"payload-pre-processing-e2e-worker-w0-59f97a-5c46c69c55-c68jw\" (UID: \"b8ac8868-81dc-42dc-aed6-22e78b580eec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-59f97a-5c46c69c55-c68jw" Nov 02 23:04:20.877218 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:20.877141 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpmvj\" (UniqueName: \"kubernetes.io/projected/83c7c9f4-adff-43e8-b6f4-0f6abf706b75-kube-api-access-fpmvj\") pod \"payload-processing-e2e-worker-w0-59f97a-578d5df469-bmvsq\" (UID: \"83c7c9f4-adff-43e8-b6f4-0f6abf706b75\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-59f97a-578d5df469-bmvsq" Nov 02 23:04:20.877362 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:20.877288 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83c7c9f4-adff-43e8-b6f4-0f6abf706b75-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-59f97a-578d5df469-bmvsq\" (UID: \"83c7c9f4-adff-43e8-b6f4-0f6abf706b75\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-59f97a-578d5df469-bmvsq" Nov 02 23:04:20.906663 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:20.906631 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-59f97a-5c46c69c55-c68jw" Nov 02 23:04:20.977998 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:20.977957 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpmvj\" (UniqueName: \"kubernetes.io/projected/83c7c9f4-adff-43e8-b6f4-0f6abf706b75-kube-api-access-fpmvj\") pod \"payload-processing-e2e-worker-w0-59f97a-578d5df469-bmvsq\" (UID: \"83c7c9f4-adff-43e8-b6f4-0f6abf706b75\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-59f97a-578d5df469-bmvsq" Nov 02 23:04:20.978151 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:20.978133 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83c7c9f4-adff-43e8-b6f4-0f6abf706b75-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-59f97a-578d5df469-bmvsq\" (UID: \"83c7c9f4-adff-43e8-b6f4-0f6abf706b75\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-59f97a-578d5df469-bmvsq" Nov 02 23:04:20.979091 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:20.979063 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83c7c9f4-adff-43e8-b6f4-0f6abf706b75-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-59f97a-578d5df469-bmvsq\" (UID: \"83c7c9f4-adff-43e8-b6f4-0f6abf706b75\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-59f97a-578d5df469-bmvsq" Nov 02 23:04:21.033021 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:21.032992 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpmvj\" (UniqueName: \"kubernetes.io/projected/83c7c9f4-adff-43e8-b6f4-0f6abf706b75-kube-api-access-fpmvj\") pod \"payload-processing-e2e-worker-w0-59f97a-578d5df469-bmvsq\" (UID: \"83c7c9f4-adff-43e8-b6f4-0f6abf706b75\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-59f97a-578d5df469-bmvsq" Nov 02 23:04:21.060867 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:21.060838 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-59f97a-578d5df469-bmvsq" Nov 02 23:04:21.234568 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:21.234531 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-db3e5cc4-698f549c7c-cjbdf"] Nov 02 23:04:21.268403 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:21.268378 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-59f97a-5c46c69c55-c68jw"] Nov 02 23:04:21.269946 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:04:21.269919 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8ac8868_81dc_42dc_aed6_22e78b580eec.slice/crio-1c66c298cfcf46f95a73868af4e0d60cc647afc9a46c1463255cfc0962c30ee6 WatchSource:0}: Error finding container 1c66c298cfcf46f95a73868af4e0d60cc647afc9a46c1463255cfc0962c30ee6: Status 404 returned error can't find the container with id 1c66c298cfcf46f95a73868af4e0d60cc647afc9a46c1463255cfc0962c30ee6 Nov 02 23:04:21.335701 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:21.335670 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-db3e5cc4-6cb868b447-8fwl7"] Nov 02 23:04:21.433303 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:21.428402 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-59f97a-578d5df469-bmvsq"] Nov 02 23:04:21.434018 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:04:21.433987 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c7c9f4_adff_43e8_b6f4_0f6abf706b75.slice/crio-d58b78ae2747a576ccbe181246e533e6427dcefd91f8e6d5f6d96bbd1890b6c8 WatchSource:0}: Error finding container d58b78ae2747a576ccbe181246e533e6427dcefd91f8e6d5f6d96bbd1890b6c8: Status 404 returned error can't find the container with id d58b78ae2747a576ccbe181246e533e6427dcefd91f8e6d5f6d96bbd1890b6c8 Nov 02 23:04:21.483672 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:21.483648 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts"] Nov 02 23:04:21.488688 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:21.488672 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts" Nov 02 23:04:21.541192 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:21.539950 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 23:04:21.541192 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:21.540085 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-b7gfc\"" Nov 02 23:04:21.541192 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:21.540116 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-14b321-serving-cert\"" Nov 02 23:04:21.542142 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:21.542114 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts"] Nov 02 23:04:21.584080 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:21.584039 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf8cc281-8664-4196-8c74-d0b92d2ae815-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts\" (UID: \"cf8cc281-8664-4196-8c74-d0b92d2ae815\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts" Nov 02 23:04:21.584080 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:21.584083 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf8cc281-8664-4196-8c74-d0b92d2ae815-metrics-certs\") pod \"maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts\" (UID: \"cf8cc281-8664-4196-8c74-d0b92d2ae815\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts" Nov 02 23:04:21.584296 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:21.584107 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf8cc281-8664-4196-8c74-d0b92d2ae815-maas-api-tls\") pod \"maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts\" (UID: \"cf8cc281-8664-4196-8c74-d0b92d2ae815\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts" Nov 02 23:04:21.584296 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:21.584238 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fsqr\" (UniqueName: \"kubernetes.io/projected/cf8cc281-8664-4196-8c74-d0b92d2ae815-kube-api-access-7fsqr\") pod \"maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts\" (UID: \"cf8cc281-8664-4196-8c74-d0b92d2ae815\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts" Nov 02 23:04:21.685106 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:21.685070 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fsqr\" (UniqueName: \"kubernetes.io/projected/cf8cc281-8664-4196-8c74-d0b92d2ae815-kube-api-access-7fsqr\") pod \"maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts\" (UID: \"cf8cc281-8664-4196-8c74-d0b92d2ae815\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts" Nov 02 23:04:21.685574 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:21.685193 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf8cc281-8664-4196-8c74-d0b92d2ae815-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts\" (UID: \"cf8cc281-8664-4196-8c74-d0b92d2ae815\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts" Nov 02 23:04:21.685574 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:21.685218 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf8cc281-8664-4196-8c74-d0b92d2ae815-metrics-certs\") pod \"maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts\" (UID: \"cf8cc281-8664-4196-8c74-d0b92d2ae815\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts" Nov 02 23:04:21.685574 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:21.685240 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf8cc281-8664-4196-8c74-d0b92d2ae815-maas-api-tls\") pod \"maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts\" (UID: \"cf8cc281-8664-4196-8c74-d0b92d2ae815\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts" Nov 02 23:04:21.685574 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:04:21.685332 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-14b321-serving-cert: secret "maas-api-e2e-worker-w3-14b321-serving-cert" not found Nov 02 23:04:21.685574 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:04:21.685391 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf8cc281-8664-4196-8c74-d0b92d2ae815-maas-api-tls podName:cf8cc281-8664-4196-8c74-d0b92d2ae815 nodeName:}" failed. No retries permitted until 2026-11-02 23:04:22.185375639 +0000 UTC m=+711.153907414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cf8cc281-8664-4196-8c74-d0b92d2ae815-maas-api-tls") pod "maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts" (UID: "cf8cc281-8664-4196-8c74-d0b92d2ae815") : secret "maas-api-e2e-worker-w3-14b321-serving-cert" not found Nov 02 23:04:21.686054 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:21.686029 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf8cc281-8664-4196-8c74-d0b92d2ae815-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts\" (UID: \"cf8cc281-8664-4196-8c74-d0b92d2ae815\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts" Nov 02 23:04:21.687601 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:21.687576 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf8cc281-8664-4196-8c74-d0b92d2ae815-metrics-certs\") pod \"maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts\" (UID: \"cf8cc281-8664-4196-8c74-d0b92d2ae815\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts" Nov 02 23:04:21.698015 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:21.697978 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-59f97a-578d5df469-bmvsq" event={"ID":"83c7c9f4-adff-43e8-b6f4-0f6abf706b75","Type":"ContainerStarted","Data":"8ef1059f836cf10efb9ff279d589fda9f2f8c047943bfe0f86c3a35c04e6e314"} Nov 02 23:04:21.698132 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:21.698032 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-59f97a-578d5df469-bmvsq" event={"ID":"83c7c9f4-adff-43e8-b6f4-0f6abf706b75","Type":"ContainerStarted","Data":"d58b78ae2747a576ccbe181246e533e6427dcefd91f8e6d5f6d96bbd1890b6c8"} Nov 02 23:04:21.698132 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:21.698088 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-59f97a-578d5df469-bmvsq" Nov 02 23:04:21.699542 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:21.699522 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-59f97a-5c46c69c55-c68jw" event={"ID":"b8ac8868-81dc-42dc-aed6-22e78b580eec","Type":"ContainerStarted","Data":"ffa14c69f3007f0390c69ec5a5b0e74ea90e692c32fbcb186a717385487edc5a"} Nov 02 23:04:21.699612 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:21.699562 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-59f97a-5c46c69c55-c68jw" event={"ID":"b8ac8868-81dc-42dc-aed6-22e78b580eec","Type":"ContainerStarted","Data":"1c66c298cfcf46f95a73868af4e0d60cc647afc9a46c1463255cfc0962c30ee6"} Nov 02 23:04:21.699612 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:21.699602 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-59f97a-5c46c69c55-c68jw" Nov 02 23:04:21.751740 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:21.750614 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fsqr\" (UniqueName: \"kubernetes.io/projected/cf8cc281-8664-4196-8c74-d0b92d2ae815-kube-api-access-7fsqr\") pod \"maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts\" (UID: \"cf8cc281-8664-4196-8c74-d0b92d2ae815\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts" Nov 02 23:04:21.806986 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:21.806929 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-59f97a-578d5df469-bmvsq" podStartSLOduration=1.806915544 podStartE2EDuration="1.806915544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:04:21.806024607 +0000 UTC m=+710.774556417" watchObservedRunningTime="2026-11-02 23:04:21.806915544 +0000 UTC m=+710.775447324" Nov 02 23:04:21.932219 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:21.924119 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-59f97a-5c46c69c55-c68jw" podStartSLOduration=1.924094668 podStartE2EDuration="1.924094668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:04:21.913435831 +0000 UTC m=+710.881967642" watchObservedRunningTime="2026-11-02 23:04:21.924094668 +0000 UTC m=+710.892626452" Nov 02 23:04:22.190213 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:22.190182 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf8cc281-8664-4196-8c74-d0b92d2ae815-maas-api-tls\") pod \"maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts\" (UID: \"cf8cc281-8664-4196-8c74-d0b92d2ae815\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts" Nov 02 23:04:22.192508 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:22.192481 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf8cc281-8664-4196-8c74-d0b92d2ae815-maas-api-tls\") pod \"maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts\" (UID: \"cf8cc281-8664-4196-8c74-d0b92d2ae815\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts" Nov 02 23:04:22.401964 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:22.401928 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts" Nov 02 23:04:22.817812 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:22.817727 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts"] Nov 02 23:04:22.822961 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:04:22.822921 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf8cc281_8664_4196_8c74_d0b92d2ae815.slice/crio-7729e5dfd4703877cf67b7f8836ec6b59ada4c2ba85bed8ee4e5af40ea01a4c3 WatchSource:0}: Error finding container 7729e5dfd4703877cf67b7f8836ec6b59ada4c2ba85bed8ee4e5af40ea01a4c3: Status 404 returned error can't find the container with id 7729e5dfd4703877cf67b7f8836ec6b59ada4c2ba85bed8ee4e5af40ea01a4c3 Nov 02 23:04:23.712162 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:23.712134 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts" event={"ID":"cf8cc281-8664-4196-8c74-d0b92d2ae815","Type":"ContainerStarted","Data":"7729e5dfd4703877cf67b7f8836ec6b59ada4c2ba85bed8ee4e5af40ea01a4c3"} Nov 02 23:04:24.718999 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:24.718959 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts" event={"ID":"cf8cc281-8664-4196-8c74-d0b92d2ae815","Type":"ContainerStarted","Data":"f438a6be9a4859bc49f30bcac35be633d26f58276a142fcf23879a2b2d14f3c4"} Nov 02 23:04:24.719387 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:24.719060 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts" Nov 02 23:04:24.832575 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:24.832511 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts" podStartSLOduration=2.946883331 podStartE2EDuration="3.83249704s" totalImagesPullingTime="885.613709ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:04:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:04:24.830715932 +0000 UTC m=+713.799247716" watchObservedRunningTime="2026-11-02 23:04:24.83249704 +0000 UTC m=+713.801028824" Nov 02 23:04:26.234951 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:26.234825 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-db3e5cc4-698f549c7c-cjbdf" podUID="a0807844-8cec-421b-b197-e1865248f554" containerName="payload-processing" containerID="cri-o://543c66524c2314c7da6298d7c03fbe93dfff8896ee6c4f64b6350f2375f0171e" gracePeriod=25 Nov 02 23:04:26.337016 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:26.336946 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-db3e5cc4-6cb868b447-8fwl7" podUID="e8efd632-03bb-452f-8da7-ffcf71621710" containerName="payload-pre-processing" containerID="cri-o://1254a90f822b92dc2291445daa4b9f53b5f5d3ed26a8e1f049d277d24f62c9f0" gracePeriod=25 Nov 02 23:04:26.491969 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:26.491878 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-db3e5cc4-698f549c7c-cjbdf" Nov 02 23:04:26.617251 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:26.617221 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-db3e5cc4-6cb868b447-8fwl7" Nov 02 23:04:26.654294 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:26.654266 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0807844-8cec-421b-b197-e1865248f554-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0807844-8cec-421b-b197-e1865248f554-plugins-config-volume\") pod \"a0807844-8cec-421b-b197-e1865248f554\" (UID: \"a0807844-8cec-421b-b197-e1865248f554\") " Nov 02 23:04:26.654439 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:26.654367 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0807844-8cec-421b-b197-e1865248f554-kube-api-access-66zx2\" (UniqueName: \"kubernetes.io/projected/a0807844-8cec-421b-b197-e1865248f554-kube-api-access-66zx2\") pod \"a0807844-8cec-421b-b197-e1865248f554\" (UID: \"a0807844-8cec-421b-b197-e1865248f554\") " Nov 02 23:04:26.654659 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:26.654635 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0807844-8cec-421b-b197-e1865248f554-plugins-config-volume" pod "a0807844-8cec-421b-b197-e1865248f554" (UID: "a0807844-8cec-421b-b197-e1865248f554"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:04:26.656417 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:26.656399 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0807844-8cec-421b-b197-e1865248f554-kube-api-access-66zx2" pod "a0807844-8cec-421b-b197-e1865248f554" (UID: "a0807844-8cec-421b-b197-e1865248f554"). InnerVolumeSpecName "kube-api-access-66zx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:04:26.732939 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:26.732877 5076 generic.go:356] "Generic (PLEG): container finished" podID="e8efd632-03bb-452f-8da7-ffcf71621710" containerID="1254a90f822b92dc2291445daa4b9f53b5f5d3ed26a8e1f049d277d24f62c9f0" exitCode=0 Nov 02 23:04:26.733103 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:26.732966 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-db3e5cc4-6cb868b447-8fwl7" Nov 02 23:04:26.733103 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:26.732964 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-db3e5cc4-6cb868b447-8fwl7" event={"ID":"e8efd632-03bb-452f-8da7-ffcf71621710","Type":"ContainerDied","Data":"1254a90f822b92dc2291445daa4b9f53b5f5d3ed26a8e1f049d277d24f62c9f0"} Nov 02 23:04:26.733243 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:26.733156 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-db3e5cc4-6cb868b447-8fwl7" event={"ID":"e8efd632-03bb-452f-8da7-ffcf71621710","Type":"ContainerDied","Data":"ea6d474c23c1e6a34af72682c8b64e04f98519fac7849cbe2d85c4bd775bfe81"} Nov 02 23:04:26.733243 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:26.733192 5076 scope.go:122] "RemoveContainer" containerID="1254a90f822b92dc2291445daa4b9f53b5f5d3ed26a8e1f049d277d24f62c9f0" Nov 02 23:04:26.734508 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:26.734462 5076 generic.go:356] "Generic (PLEG): container finished" podID="a0807844-8cec-421b-b197-e1865248f554" containerID="543c66524c2314c7da6298d7c03fbe93dfff8896ee6c4f64b6350f2375f0171e" exitCode=0 Nov 02 23:04:26.734581 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:26.734505 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-db3e5cc4-698f549c7c-cjbdf" event={"ID":"a0807844-8cec-421b-b197-e1865248f554","Type":"ContainerDied","Data":"543c66524c2314c7da6298d7c03fbe93dfff8896ee6c4f64b6350f2375f0171e"} Nov 02 23:04:26.734581 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:26.734511 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-db3e5cc4-698f549c7c-cjbdf" Nov 02 23:04:26.734581 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:26.734537 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-db3e5cc4-698f549c7c-cjbdf" event={"ID":"a0807844-8cec-421b-b197-e1865248f554","Type":"ContainerDied","Data":"cf4d6edcbe231966afda22ad39300394f038ac67be2d9ab5db15d8613e5de98a"} Nov 02 23:04:26.744722 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:26.744702 5076 scope.go:122] "RemoveContainer" containerID="1254a90f822b92dc2291445daa4b9f53b5f5d3ed26a8e1f049d277d24f62c9f0" Nov 02 23:04:26.745085 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:04:26.745062 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1254a90f822b92dc2291445daa4b9f53b5f5d3ed26a8e1f049d277d24f62c9f0\": container with ID starting with 1254a90f822b92dc2291445daa4b9f53b5f5d3ed26a8e1f049d277d24f62c9f0 not found: ID does not exist" containerID="1254a90f822b92dc2291445daa4b9f53b5f5d3ed26a8e1f049d277d24f62c9f0" Nov 02 23:04:26.745178 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:26.745093 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1254a90f822b92dc2291445daa4b9f53b5f5d3ed26a8e1f049d277d24f62c9f0"} err="failed to get container status \"1254a90f822b92dc2291445daa4b9f53b5f5d3ed26a8e1f049d277d24f62c9f0\": rpc error: code = NotFound desc = could not find container \"1254a90f822b92dc2291445daa4b9f53b5f5d3ed26a8e1f049d277d24f62c9f0\": container with ID starting with 1254a90f822b92dc2291445daa4b9f53b5f5d3ed26a8e1f049d277d24f62c9f0 not found: ID does not exist" Nov 02 23:04:26.745178 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:26.745111 5076 scope.go:122] "RemoveContainer" containerID="543c66524c2314c7da6298d7c03fbe93dfff8896ee6c4f64b6350f2375f0171e" Nov 02 23:04:26.754784 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:26.754759 5076 scope.go:122] "RemoveContainer" containerID="543c66524c2314c7da6298d7c03fbe93dfff8896ee6c4f64b6350f2375f0171e" Nov 02 23:04:26.755160 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:26.755132 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8efd632-03bb-452f-8da7-ffcf71621710-kube-api-access-2kfg8\" (UniqueName: \"kubernetes.io/projected/e8efd632-03bb-452f-8da7-ffcf71621710-kube-api-access-2kfg8\") pod \"e8efd632-03bb-452f-8da7-ffcf71621710\" (UID: \"e8efd632-03bb-452f-8da7-ffcf71621710\") " Nov 02 23:04:26.755251 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:04:26.755161 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"543c66524c2314c7da6298d7c03fbe93dfff8896ee6c4f64b6350f2375f0171e\": container with ID starting with 543c66524c2314c7da6298d7c03fbe93dfff8896ee6c4f64b6350f2375f0171e not found: ID does not exist" containerID="543c66524c2314c7da6298d7c03fbe93dfff8896ee6c4f64b6350f2375f0171e" Nov 02 23:04:26.755251 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:26.755206 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"543c66524c2314c7da6298d7c03fbe93dfff8896ee6c4f64b6350f2375f0171e"} err="failed to get container status \"543c66524c2314c7da6298d7c03fbe93dfff8896ee6c4f64b6350f2375f0171e\": rpc error: code = NotFound desc = could not find container \"543c66524c2314c7da6298d7c03fbe93dfff8896ee6c4f64b6350f2375f0171e\": container with ID starting with 543c66524c2314c7da6298d7c03fbe93dfff8896ee6c4f64b6350f2375f0171e not found: ID does not exist" Nov 02 23:04:26.755251 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:26.755179 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8efd632-03bb-452f-8da7-ffcf71621710-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8efd632-03bb-452f-8da7-ffcf71621710-plugins-config-volume\") pod \"e8efd632-03bb-452f-8da7-ffcf71621710\" (UID: \"e8efd632-03bb-452f-8da7-ffcf71621710\") " Nov 02 23:04:26.755607 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:26.755570 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8efd632-03bb-452f-8da7-ffcf71621710-plugins-config-volume" pod "e8efd632-03bb-452f-8da7-ffcf71621710" (UID: "e8efd632-03bb-452f-8da7-ffcf71621710"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:04:26.755770 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:26.755752 5076 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0807844-8cec-421b-b197-e1865248f554-plugins-config-volume\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:04:26.755814 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:26.755777 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66zx2\" (UniqueName: \"kubernetes.io/projected/a0807844-8cec-421b-b197-e1865248f554-kube-api-access-66zx2\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:04:26.755814 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:26.755791 5076 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8efd632-03bb-452f-8da7-ffcf71621710-plugins-config-volume\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:04:26.757154 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:26.757134 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8efd632-03bb-452f-8da7-ffcf71621710-kube-api-access-2kfg8" pod "e8efd632-03bb-452f-8da7-ffcf71621710" (UID: "e8efd632-03bb-452f-8da7-ffcf71621710"). InnerVolumeSpecName "kube-api-access-2kfg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:04:26.857166 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:26.857133 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kfg8\" (UniqueName: \"kubernetes.io/projected/e8efd632-03bb-452f-8da7-ffcf71621710-kube-api-access-2kfg8\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:04:26.889275 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:26.889244 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-db3e5cc4-698f549c7c-cjbdf"] Nov 02 23:04:26.891180 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:26.891158 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-db3e5cc4-698f549c7c-cjbdf"] Nov 02 23:04:27.147707 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:27.147672 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0807844-8cec-421b-b197-e1865248f554" path="/var/lib/kubelet/pods/a0807844-8cec-421b-b197-e1865248f554/volumes" Nov 02 23:04:27.147982 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:27.147968 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-db3e5cc4-6cb868b447-8fwl7"] Nov 02 23:04:27.152117 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:27.152093 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-db3e5cc4-6cb868b447-8fwl7"] Nov 02 23:04:29.147109 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:29.147075 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8efd632-03bb-452f-8da7-ffcf71621710" path="/var/lib/kubelet/pods/e8efd632-03bb-452f-8da7-ffcf71621710/volumes" Nov 02 23:04:30.737290 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:30.737258 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts" Nov 02 23:04:30.779171 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:30.779132 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p"] Nov 02 23:04:30.779635 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:30.779617 5076 cpu_manager.go:414] "containerMap: removing container" podUID="a0807844-8cec-421b-b197-e1865248f554" containerName="payload-processing" Nov 02 23:04:30.779720 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:30.779635 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0807844-8cec-421b-b197-e1865248f554" containerName="payload-processing" Nov 02 23:04:30.779720 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:30.779669 5076 cpu_manager.go:414] "containerMap: removing container" podUID="e8efd632-03bb-452f-8da7-ffcf71621710" containerName="payload-pre-processing" Nov 02 23:04:30.779720 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:30.779677 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8efd632-03bb-452f-8da7-ffcf71621710" containerName="payload-pre-processing" Nov 02 23:04:30.779876 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:30.779810 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8efd632-03bb-452f-8da7-ffcf71621710" containerName="payload-pre-processing" Nov 02 23:04:30.779876 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:30.779829 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0807844-8cec-421b-b197-e1865248f554" containerName="payload-processing" Nov 02 23:04:30.784256 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:30.784235 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p" Nov 02 23:04:30.796399 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:30.796378 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b982eab-939c-49d4-87ec-b6b9a07a3cbc-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p\" (UID: \"3b982eab-939c-49d4-87ec-b6b9a07a3cbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p" Nov 02 23:04:30.796490 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:30.796407 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b982eab-939c-49d4-87ec-b6b9a07a3cbc-maas-api-tls\") pod \"maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p\" (UID: \"3b982eab-939c-49d4-87ec-b6b9a07a3cbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p" Nov 02 23:04:30.796490 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:30.796427 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b982eab-939c-49d4-87ec-b6b9a07a3cbc-metrics-certs\") pod \"maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p\" (UID: \"3b982eab-939c-49d4-87ec-b6b9a07a3cbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p" Nov 02 23:04:30.796708 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:30.796685 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djgrw\" (UniqueName: \"kubernetes.io/projected/3b982eab-939c-49d4-87ec-b6b9a07a3cbc-kube-api-access-djgrw\") pod \"maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p\" (UID: \"3b982eab-939c-49d4-87ec-b6b9a07a3cbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p" Nov 02 23:04:30.834580 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:30.834546 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-59f97a-serving-cert\"" Nov 02 23:04:30.837745 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:30.837705 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p"] Nov 02 23:04:30.898335 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:30.898303 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b982eab-939c-49d4-87ec-b6b9a07a3cbc-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p\" (UID: \"3b982eab-939c-49d4-87ec-b6b9a07a3cbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p" Nov 02 23:04:30.898543 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:30.898345 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b982eab-939c-49d4-87ec-b6b9a07a3cbc-maas-api-tls\") pod \"maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p\" (UID: \"3b982eab-939c-49d4-87ec-b6b9a07a3cbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p" Nov 02 23:04:30.898543 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:30.898371 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b982eab-939c-49d4-87ec-b6b9a07a3cbc-metrics-certs\") pod \"maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p\" (UID: \"3b982eab-939c-49d4-87ec-b6b9a07a3cbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p" Nov 02 23:04:30.898543 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:30.898418 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djgrw\" (UniqueName: \"kubernetes.io/projected/3b982eab-939c-49d4-87ec-b6b9a07a3cbc-kube-api-access-djgrw\") pod \"maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p\" (UID: \"3b982eab-939c-49d4-87ec-b6b9a07a3cbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p" Nov 02 23:04:30.898543 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:04:30.898459 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-59f97a-serving-cert: secret "maas-api-e2e-worker-w0-59f97a-serving-cert" not found Nov 02 23:04:30.898543 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:04:30.898531 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b982eab-939c-49d4-87ec-b6b9a07a3cbc-maas-api-tls podName:3b982eab-939c-49d4-87ec-b6b9a07a3cbc nodeName:}" failed. No retries permitted until 2026-11-02 23:04:31.398511932 +0000 UTC m=+720.367043695 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3b982eab-939c-49d4-87ec-b6b9a07a3cbc-maas-api-tls") pod "maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p" (UID: "3b982eab-939c-49d4-87ec-b6b9a07a3cbc") : secret "maas-api-e2e-worker-w0-59f97a-serving-cert" not found Nov 02 23:04:30.899092 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:30.899066 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b982eab-939c-49d4-87ec-b6b9a07a3cbc-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p\" (UID: \"3b982eab-939c-49d4-87ec-b6b9a07a3cbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p" Nov 02 23:04:30.901598 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:30.901570 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b982eab-939c-49d4-87ec-b6b9a07a3cbc-metrics-certs\") pod \"maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p\" (UID: \"3b982eab-939c-49d4-87ec-b6b9a07a3cbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p" Nov 02 23:04:30.953909 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:30.953850 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djgrw\" (UniqueName: \"kubernetes.io/projected/3b982eab-939c-49d4-87ec-b6b9a07a3cbc-kube-api-access-djgrw\") pod \"maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p\" (UID: \"3b982eab-939c-49d4-87ec-b6b9a07a3cbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p" Nov 02 23:04:31.403636 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:31.403604 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b982eab-939c-49d4-87ec-b6b9a07a3cbc-maas-api-tls\") pod \"maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p\" (UID: \"3b982eab-939c-49d4-87ec-b6b9a07a3cbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p" Nov 02 23:04:31.405947 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:31.405922 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b982eab-939c-49d4-87ec-b6b9a07a3cbc-maas-api-tls\") pod \"maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p\" (UID: \"3b982eab-939c-49d4-87ec-b6b9a07a3cbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p" Nov 02 23:04:31.697702 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:31.697672 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p" Nov 02 23:04:32.091404 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:32.091369 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p"] Nov 02 23:04:32.091827 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:04:32.091562 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b982eab_939c_49d4_87ec_b6b9a07a3cbc.slice/crio-6428135c69adfa69aabf46b146a3ae4bdee8456958674b6ed65dc90d435b11ca WatchSource:0}: Error finding container 6428135c69adfa69aabf46b146a3ae4bdee8456958674b6ed65dc90d435b11ca: Status 404 returned error can't find the container with id 6428135c69adfa69aabf46b146a3ae4bdee8456958674b6ed65dc90d435b11ca Nov 02 23:04:32.708288 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:32.708259 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-59f97a-5c46c69c55-c68jw" Nov 02 23:04:32.708470 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:32.708455 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-59f97a-578d5df469-bmvsq" Nov 02 23:04:32.773091 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:32.773058 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p" event={"ID":"3b982eab-939c-49d4-87ec-b6b9a07a3cbc","Type":"ContainerStarted","Data":"6428135c69adfa69aabf46b146a3ae4bdee8456958674b6ed65dc90d435b11ca"} Nov 02 23:04:33.780341 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:33.780301 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p" event={"ID":"3b982eab-939c-49d4-87ec-b6b9a07a3cbc","Type":"ContainerStarted","Data":"040852454c25987f1a6d78ad2adccee22518eef86784b376238c4445f6ed5564"} Nov 02 23:04:33.780756 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:33.780443 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p" Nov 02 23:04:33.895027 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:33.894976 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p" podStartSLOduration=3.049460357 podStartE2EDuration="3.894953397s" totalImagesPullingTime="845.49304ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:04:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:04:33.893095506 +0000 UTC m=+722.861627291" watchObservedRunningTime="2026-11-02 23:04:33.894953397 +0000 UTC m=+722.863485215" Nov 02 23:04:38.139914 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:38.134114 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bdaf93e3-nhjqv"] Nov 02 23:04:38.190760 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:38.190721 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bdaf93e3-nhjqv"] Nov 02 23:04:38.848027 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:38.830347 5076 generic.go:356] "Generic (PLEG): container finished" podID="f6b0b2ba-6e13-4d79-b2b1-243f80b5703c" containerID="e1f08f0a7fc6972d298961353624a3625a1ae5eeaea913dabd22615b1b391dd9" exitCode=0 Nov 02 23:04:38.848027 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:38.830523 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" event={"ID":"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c","Type":"ContainerDied","Data":"e1f08f0a7fc6972d298961353624a3625a1ae5eeaea913dabd22615b1b391dd9"} Nov 02 23:04:38.848027 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:38.831020 5076 scope.go:122] "RemoveContainer" containerID="e1f08f0a7fc6972d298961353624a3625a1ae5eeaea913dabd22615b1b391dd9" Nov 02 23:04:38.861910 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:38.850410 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:04:38.861910 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:38.856992 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:04:38.861910 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:38.857050 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:04:39.170747 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:39.170257 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f" path="/var/lib/kubelet/pods/fa743fb7-1874-4f16-bb9b-e7c31fc0fd7f/volumes" Nov 02 23:04:39.792728 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:39.792702 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p" Nov 02 23:04:39.851934 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:39.851509 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" event={"ID":"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c","Type":"ContainerStarted","Data":"fd114a8aad01ca77a63c98e495907d5bf45fbbaa37c4200ee7dae82f1572c54e"} Nov 02 23:04:39.887712 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:04:39.887664 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:04:40.748632 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:40.748224 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" Nov 02 23:04:40.748632 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:40.748339 5076 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 02 23:04:40.748632 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:40.748392 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" podUID="f6b0b2ba-6e13-4d79-b2b1-243f80b5703c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 02 23:04:41.751427 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:41.751380 5076 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 02 23:04:41.752106 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:41.752073 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" podUID="f6b0b2ba-6e13-4d79-b2b1-243f80b5703c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 02 23:04:42.748746 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:42.748262 5076 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 02 23:04:42.748746 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:42.748324 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" podUID="f6b0b2ba-6e13-4d79-b2b1-243f80b5703c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 02 23:04:43.746879 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:43.746832 5076 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 02 23:04:43.747307 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:43.746947 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" podUID="f6b0b2ba-6e13-4d79-b2b1-243f80b5703c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 02 23:04:44.747581 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:44.747535 5076 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 02 23:04:44.748122 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:44.747612 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" podUID="f6b0b2ba-6e13-4d79-b2b1-243f80b5703c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 02 23:04:45.750780 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:45.750749 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" Nov 02 23:04:45.751297 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:45.751135 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" Nov 02 23:04:45.751942 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:45.751926 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" Nov 02 23:04:46.902681 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:46.902596 5076 generic.go:356] "Generic (PLEG): container finished" podID="cc003675-8adf-4c2d-af38-9e3ad5af69f4" containerID="8330bd727513640206cf1fe03726eba7c69976ae3a5bdfec0f9e67bd2ce5254d" exitCode=0 Nov 02 23:04:46.903151 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:46.902680 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" event={"ID":"cc003675-8adf-4c2d-af38-9e3ad5af69f4","Type":"ContainerDied","Data":"8330bd727513640206cf1fe03726eba7c69976ae3a5bdfec0f9e67bd2ce5254d"} Nov 02 23:04:46.903315 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:46.903298 5076 scope.go:122] "RemoveContainer" containerID="8330bd727513640206cf1fe03726eba7c69976ae3a5bdfec0f9e67bd2ce5254d" Nov 02 23:04:46.904512 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:46.904483 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:04:46.904599 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:46.904551 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:04:46.904599 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:46.904587 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:04:47.910205 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:47.910165 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" event={"ID":"cc003675-8adf-4c2d-af38-9e3ad5af69f4","Type":"ContainerStarted","Data":"619f650a70fa12a4f1bce5234bde4b9556d72ed621420323781af0d31a62b115"} Nov 02 23:04:48.160579 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:48.160486 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" Nov 02 23:04:48.160815 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:48.160791 5076 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 02 23:04:48.160884 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:48.160863 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" podUID="cc003675-8adf-4c2d-af38-9e3ad5af69f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 02 23:04:49.161061 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:49.161022 5076 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 02 23:04:49.161469 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:49.161086 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" podUID="cc003675-8adf-4c2d-af38-9e3ad5af69f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 02 23:04:50.161265 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:50.161224 5076 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 02 23:04:50.161661 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:50.161293 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" podUID="cc003675-8adf-4c2d-af38-9e3ad5af69f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 02 23:04:51.160521 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:51.160483 5076 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 02 23:04:51.160729 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:51.160537 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" podUID="cc003675-8adf-4c2d-af38-9e3ad5af69f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 02 23:04:52.161050 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:52.161018 5076 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 02 23:04:52.161463 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:52.161071 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" podUID="cc003675-8adf-4c2d-af38-9e3ad5af69f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 02 23:04:53.164677 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:53.164641 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" Nov 02 23:04:53.165193 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:53.164990 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" Nov 02 23:04:53.165760 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:53.165739 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" Nov 02 23:04:58.600689 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:58.600648 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw"] Nov 02 23:04:58.604921 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:58.604884 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw" Nov 02 23:04:58.630711 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:58.630664 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/365ba2a8-0d25-4d3e-b6b0-eb577be48fcd-maas-api-tls\") pod \"maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw\" (UID: \"365ba2a8-0d25-4d3e-b6b0-eb577be48fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw" Nov 02 23:04:58.631009 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:58.630975 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf6ms\" (UniqueName: \"kubernetes.io/projected/365ba2a8-0d25-4d3e-b6b0-eb577be48fcd-kube-api-access-jf6ms\") pod \"maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw\" (UID: \"365ba2a8-0d25-4d3e-b6b0-eb577be48fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw" Nov 02 23:04:58.631281 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:58.631262 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/365ba2a8-0d25-4d3e-b6b0-eb577be48fcd-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw\" (UID: \"365ba2a8-0d25-4d3e-b6b0-eb577be48fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw" Nov 02 23:04:58.631388 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:58.631331 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/365ba2a8-0d25-4d3e-b6b0-eb577be48fcd-metrics-certs\") pod \"maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw\" (UID: \"365ba2a8-0d25-4d3e-b6b0-eb577be48fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw" Nov 02 23:04:58.661793 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:58.661759 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw"] Nov 02 23:04:58.706084 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:58.706048 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-ba1918-serving-cert\"" Nov 02 23:04:58.731908 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:58.731861 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/365ba2a8-0d25-4d3e-b6b0-eb577be48fcd-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw\" (UID: \"365ba2a8-0d25-4d3e-b6b0-eb577be48fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw" Nov 02 23:04:58.732082 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:58.731940 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/365ba2a8-0d25-4d3e-b6b0-eb577be48fcd-metrics-certs\") pod \"maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw\" (UID: \"365ba2a8-0d25-4d3e-b6b0-eb577be48fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw" Nov 02 23:04:58.732082 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:58.731995 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/365ba2a8-0d25-4d3e-b6b0-eb577be48fcd-maas-api-tls\") pod \"maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw\" (UID: \"365ba2a8-0d25-4d3e-b6b0-eb577be48fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw" Nov 02 23:04:58.732082 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:58.732032 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf6ms\" (UniqueName: \"kubernetes.io/projected/365ba2a8-0d25-4d3e-b6b0-eb577be48fcd-kube-api-access-jf6ms\") pod \"maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw\" (UID: \"365ba2a8-0d25-4d3e-b6b0-eb577be48fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw" Nov 02 23:04:58.732383 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:04:58.732361 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-ba1918-serving-cert: secret "maas-api-e2e-worker-w2-ba1918-serving-cert" not found Nov 02 23:04:58.732567 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:04:58.732553 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/365ba2a8-0d25-4d3e-b6b0-eb577be48fcd-maas-api-tls podName:365ba2a8-0d25-4d3e-b6b0-eb577be48fcd nodeName:}" failed. No retries permitted until 2026-11-02 23:04:59.232532331 +0000 UTC m=+748.201064097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/365ba2a8-0d25-4d3e-b6b0-eb577be48fcd-maas-api-tls") pod "maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw" (UID: "365ba2a8-0d25-4d3e-b6b0-eb577be48fcd") : secret "maas-api-e2e-worker-w2-ba1918-serving-cert" not found Nov 02 23:04:58.732823 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:58.732802 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/365ba2a8-0d25-4d3e-b6b0-eb577be48fcd-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw\" (UID: \"365ba2a8-0d25-4d3e-b6b0-eb577be48fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw" Nov 02 23:04:58.734565 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:58.734544 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/365ba2a8-0d25-4d3e-b6b0-eb577be48fcd-metrics-certs\") pod \"maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw\" (UID: \"365ba2a8-0d25-4d3e-b6b0-eb577be48fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw" Nov 02 23:04:58.789582 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:58.789547 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf6ms\" (UniqueName: \"kubernetes.io/projected/365ba2a8-0d25-4d3e-b6b0-eb577be48fcd-kube-api-access-jf6ms\") pod \"maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw\" (UID: \"365ba2a8-0d25-4d3e-b6b0-eb577be48fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw" Nov 02 23:04:59.241410 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:59.241374 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/365ba2a8-0d25-4d3e-b6b0-eb577be48fcd-maas-api-tls\") pod \"maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw\" (UID: \"365ba2a8-0d25-4d3e-b6b0-eb577be48fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw" Nov 02 23:04:59.243819 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:59.243786 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/365ba2a8-0d25-4d3e-b6b0-eb577be48fcd-maas-api-tls\") pod \"maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw\" (UID: \"365ba2a8-0d25-4d3e-b6b0-eb577be48fcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw" Nov 02 23:04:59.538047 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:59.537961 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw" Nov 02 23:04:59.904117 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:04:59.904085 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw"] Nov 02 23:04:59.904981 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:04:59.904957 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod365ba2a8_0d25_4d3e_b6b0_eb577be48fcd.slice/crio-412b10dd52afc6346502102ef3125058c21c59d0ddbca28a037970711c97057c WatchSource:0}: Error finding container 412b10dd52afc6346502102ef3125058c21c59d0ddbca28a037970711c97057c: Status 404 returned error can't find the container with id 412b10dd52afc6346502102ef3125058c21c59d0ddbca28a037970711c97057c Nov 02 23:05:00.004157 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:05:00.004117 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw" event={"ID":"365ba2a8-0d25-4d3e-b6b0-eb577be48fcd","Type":"ContainerStarted","Data":"412b10dd52afc6346502102ef3125058c21c59d0ddbca28a037970711c97057c"} Nov 02 23:05:01.010270 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:05:01.010172 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw" event={"ID":"365ba2a8-0d25-4d3e-b6b0-eb577be48fcd","Type":"ContainerStarted","Data":"d15ef001caec8a7c00633e7f071acb6486374af60e922b63a650b3791a01421c"} Nov 02 23:05:01.010270 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:05:01.010243 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw" Nov 02 23:05:01.119413 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:05:01.119355 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw" podStartSLOduration=2.2711477159999998 podStartE2EDuration="3.119339784s" totalImagesPullingTime="848.192068ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:04:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:05:01.118813237 +0000 UTC m=+750.087345022" watchObservedRunningTime="2026-11-02 23:05:01.119339784 +0000 UTC m=+750.087871568" Nov 02 23:05:07.021036 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:05:07.021004 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw" Nov 02 23:05:09.883104 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:05:09.883064 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:05:12.465813 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:05:12.465719 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4"] Nov 02 23:05:12.470661 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:05:12.470635 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4" Nov 02 23:05:12.522082 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:05:12.522055 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4"] Nov 02 23:05:12.566993 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:05:12.566967 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-fb6e1d-serving-cert\"" Nov 02 23:05:12.612755 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:05:12.612717 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7485875c-b5a6-4532-833e-80cc5741b3ed-maas-api-tls\") pod \"maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4\" (UID: \"7485875c-b5a6-4532-833e-80cc5741b3ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4" Nov 02 23:05:12.612927 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:05:12.612793 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7485875c-b5a6-4532-833e-80cc5741b3ed-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4\" (UID: \"7485875c-b5a6-4532-833e-80cc5741b3ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4" Nov 02 23:05:12.612927 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:05:12.612922 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7485875c-b5a6-4532-833e-80cc5741b3ed-metrics-certs\") pod \"maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4\" (UID: \"7485875c-b5a6-4532-833e-80cc5741b3ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4" Nov 02 23:05:12.613011 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:05:12.612956 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbh2b\" (UniqueName: \"kubernetes.io/projected/7485875c-b5a6-4532-833e-80cc5741b3ed-kube-api-access-mbh2b\") pod \"maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4\" (UID: \"7485875c-b5a6-4532-833e-80cc5741b3ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4" Nov 02 23:05:12.716844 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:05:12.716747 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7485875c-b5a6-4532-833e-80cc5741b3ed-maas-api-tls\") pod \"maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4\" (UID: \"7485875c-b5a6-4532-833e-80cc5741b3ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4" Nov 02 23:05:12.718673 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:05:12.717167 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7485875c-b5a6-4532-833e-80cc5741b3ed-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4\" (UID: \"7485875c-b5a6-4532-833e-80cc5741b3ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4" Nov 02 23:05:12.718673 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:05:12.718338 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-fb6e1d-serving-cert: secret "maas-api-e2e-worker-w4-fb6e1d-serving-cert" not found Nov 02 23:05:12.718673 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:05:12.718417 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7485875c-b5a6-4532-833e-80cc5741b3ed-maas-api-tls podName:7485875c-b5a6-4532-833e-80cc5741b3ed nodeName:}" failed. No retries permitted until 2026-11-02 23:05:13.218397181 +0000 UTC m=+762.186928958 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7485875c-b5a6-4532-833e-80cc5741b3ed-maas-api-tls") pod "maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4" (UID: "7485875c-b5a6-4532-833e-80cc5741b3ed") : secret "maas-api-e2e-worker-w4-fb6e1d-serving-cert" not found Nov 02 23:05:12.719088 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:05:12.718993 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7485875c-b5a6-4532-833e-80cc5741b3ed-metrics-certs\") pod \"maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4\" (UID: \"7485875c-b5a6-4532-833e-80cc5741b3ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4" Nov 02 23:05:12.719504 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:05:12.719210 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbh2b\" (UniqueName: \"kubernetes.io/projected/7485875c-b5a6-4532-833e-80cc5741b3ed-kube-api-access-mbh2b\") pod \"maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4\" (UID: \"7485875c-b5a6-4532-833e-80cc5741b3ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4" Nov 02 23:05:12.720647 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:05:12.720617 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7485875c-b5a6-4532-833e-80cc5741b3ed-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4\" (UID: \"7485875c-b5a6-4532-833e-80cc5741b3ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4" Nov 02 23:05:12.727634 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:05:12.727603 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7485875c-b5a6-4532-833e-80cc5741b3ed-metrics-certs\") pod \"maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4\" (UID: \"7485875c-b5a6-4532-833e-80cc5741b3ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4" Nov 02 23:05:12.776574 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:05:12.776534 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbh2b\" (UniqueName: \"kubernetes.io/projected/7485875c-b5a6-4532-833e-80cc5741b3ed-kube-api-access-mbh2b\") pod \"maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4\" (UID: \"7485875c-b5a6-4532-833e-80cc5741b3ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4" Nov 02 23:05:13.224935 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:05:13.224864 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7485875c-b5a6-4532-833e-80cc5741b3ed-maas-api-tls\") pod \"maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4\" (UID: \"7485875c-b5a6-4532-833e-80cc5741b3ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4" Nov 02 23:05:13.227470 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:05:13.227438 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7485875c-b5a6-4532-833e-80cc5741b3ed-maas-api-tls\") pod \"maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4\" (UID: \"7485875c-b5a6-4532-833e-80cc5741b3ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4" Nov 02 23:05:13.384035 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:05:13.383987 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4" Nov 02 23:05:13.766420 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:05:13.766387 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4"] Nov 02 23:05:13.768776 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:05:13.768732 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7485875c_b5a6_4532_833e_80cc5741b3ed.slice/crio-64e87a64bb843525ee908275fff2ee1b6cf7f24e5422ca498da6df8727f0b662 WatchSource:0}: Error finding container 64e87a64bb843525ee908275fff2ee1b6cf7f24e5422ca498da6df8727f0b662: Status 404 returned error can't find the container with id 64e87a64bb843525ee908275fff2ee1b6cf7f24e5422ca498da6df8727f0b662 Nov 02 23:05:14.080090 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:05:14.080003 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4" event={"ID":"7485875c-b5a6-4532-833e-80cc5741b3ed","Type":"ContainerStarted","Data":"64e87a64bb843525ee908275fff2ee1b6cf7f24e5422ca498da6df8727f0b662"} Nov 02 23:05:15.086499 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:05:15.086459 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4" event={"ID":"7485875c-b5a6-4532-833e-80cc5741b3ed","Type":"ContainerStarted","Data":"85391d186a6e7276e8491285ade9db1ad12cd78c9184aee5253a94fa0d446d44"} Nov 02 23:05:15.086919 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:05:15.086529 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4" Nov 02 23:05:15.196600 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:05:15.196548 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4" podStartSLOduration=2.365097729 podStartE2EDuration="3.196534073s" totalImagesPullingTime="831.436344ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:05:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:05:15.195025221 +0000 UTC m=+764.163557006" watchObservedRunningTime="2026-11-02 23:05:15.196534073 +0000 UTC m=+764.165065856" Nov 02 23:05:21.097934 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:05:21.097877 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4" Nov 02 23:05:39.884438 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:05:39.884401 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:06:09.883998 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:06:09.883956 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:06:39.892827 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:06:39.892791 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:07:09.882783 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:07:09.882689 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:07:19.376826 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:19.376776 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv"] Nov 02 23:07:19.381616 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:19.381579 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" Nov 02 23:07:19.433525 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:19.433486 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv"] Nov 02 23:07:19.477101 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:19.477069 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-fb6e1d\"/\"kube-root-ca.crt\"" Nov 02 23:07:19.477101 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:19.477069 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-fb6e1d\"/\"openshift-service-ca.crt\"" Nov 02 23:07:19.478299 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:19.478277 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-fb6e1d\"/\"default-dockercfg-66dv6\"" Nov 02 23:07:19.478356 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:19.478336 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-fb6e1d\"/\"premium-w4-fb6e1d-kserve-self-signed-certs\"" Nov 02 23:07:19.506692 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:19.506646 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfxw6\" (UniqueName: \"kubernetes.io/projected/c5cd8831-584f-47bb-9580-4d41deab29dc-kube-api-access-xfxw6\") pod \"premium-w4-fb6e1d-kserve-64d7869d66-8rzmv\" (UID: \"c5cd8831-584f-47bb-9580-4d41deab29dc\") " pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" Nov 02 23:07:19.506692 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:19.506695 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5cd8831-584f-47bb-9580-4d41deab29dc-model-cache\") pod \"premium-w4-fb6e1d-kserve-64d7869d66-8rzmv\" (UID: \"c5cd8831-584f-47bb-9580-4d41deab29dc\") " pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" Nov 02 23:07:19.506963 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:19.506718 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5cd8831-584f-47bb-9580-4d41deab29dc-home\") pod \"premium-w4-fb6e1d-kserve-64d7869d66-8rzmv\" (UID: \"c5cd8831-584f-47bb-9580-4d41deab29dc\") " pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" Nov 02 23:07:19.506963 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:19.506841 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5cd8831-584f-47bb-9580-4d41deab29dc-tls-certs\") pod \"premium-w4-fb6e1d-kserve-64d7869d66-8rzmv\" (UID: \"c5cd8831-584f-47bb-9580-4d41deab29dc\") " pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" Nov 02 23:07:19.506963 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:19.506952 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5cd8831-584f-47bb-9580-4d41deab29dc-tmp-dir\") pod \"premium-w4-fb6e1d-kserve-64d7869d66-8rzmv\" (UID: \"c5cd8831-584f-47bb-9580-4d41deab29dc\") " pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" Nov 02 23:07:19.507075 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:19.506993 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5cd8831-584f-47bb-9580-4d41deab29dc-dshm\") pod \"premium-w4-fb6e1d-kserve-64d7869d66-8rzmv\" (UID: \"c5cd8831-584f-47bb-9580-4d41deab29dc\") " pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" Nov 02 23:07:19.607877 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:19.607839 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfxw6\" (UniqueName: \"kubernetes.io/projected/c5cd8831-584f-47bb-9580-4d41deab29dc-kube-api-access-xfxw6\") pod \"premium-w4-fb6e1d-kserve-64d7869d66-8rzmv\" (UID: \"c5cd8831-584f-47bb-9580-4d41deab29dc\") " pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" Nov 02 23:07:19.607877 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:19.607878 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5cd8831-584f-47bb-9580-4d41deab29dc-model-cache\") pod \"premium-w4-fb6e1d-kserve-64d7869d66-8rzmv\" (UID: \"c5cd8831-584f-47bb-9580-4d41deab29dc\") " pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" Nov 02 23:07:19.608161 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:19.607948 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5cd8831-584f-47bb-9580-4d41deab29dc-home\") pod \"premium-w4-fb6e1d-kserve-64d7869d66-8rzmv\" (UID: \"c5cd8831-584f-47bb-9580-4d41deab29dc\") " pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" Nov 02 23:07:19.608161 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:19.607989 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5cd8831-584f-47bb-9580-4d41deab29dc-tls-certs\") pod \"premium-w4-fb6e1d-kserve-64d7869d66-8rzmv\" (UID: \"c5cd8831-584f-47bb-9580-4d41deab29dc\") " pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" Nov 02 23:07:19.608161 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:19.608027 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5cd8831-584f-47bb-9580-4d41deab29dc-tmp-dir\") pod \"premium-w4-fb6e1d-kserve-64d7869d66-8rzmv\" (UID: \"c5cd8831-584f-47bb-9580-4d41deab29dc\") " pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" Nov 02 23:07:19.608161 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:19.608119 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5cd8831-584f-47bb-9580-4d41deab29dc-dshm\") pod \"premium-w4-fb6e1d-kserve-64d7869d66-8rzmv\" (UID: \"c5cd8831-584f-47bb-9580-4d41deab29dc\") " pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" Nov 02 23:07:19.608384 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:19.608364 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5cd8831-584f-47bb-9580-4d41deab29dc-model-cache\") pod \"premium-w4-fb6e1d-kserve-64d7869d66-8rzmv\" (UID: \"c5cd8831-584f-47bb-9580-4d41deab29dc\") " pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" Nov 02 23:07:19.608443 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:19.608373 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5cd8831-584f-47bb-9580-4d41deab29dc-tmp-dir\") pod \"premium-w4-fb6e1d-kserve-64d7869d66-8rzmv\" (UID: \"c5cd8831-584f-47bb-9580-4d41deab29dc\") " pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" Nov 02 23:07:19.608487 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:19.608438 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5cd8831-584f-47bb-9580-4d41deab29dc-home\") pod \"premium-w4-fb6e1d-kserve-64d7869d66-8rzmv\" (UID: \"c5cd8831-584f-47bb-9580-4d41deab29dc\") " pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" Nov 02 23:07:19.610249 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:19.610229 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5cd8831-584f-47bb-9580-4d41deab29dc-dshm\") pod \"premium-w4-fb6e1d-kserve-64d7869d66-8rzmv\" (UID: \"c5cd8831-584f-47bb-9580-4d41deab29dc\") " pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" Nov 02 23:07:19.610510 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:19.610493 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5cd8831-584f-47bb-9580-4d41deab29dc-tls-certs\") pod \"premium-w4-fb6e1d-kserve-64d7869d66-8rzmv\" (UID: \"c5cd8831-584f-47bb-9580-4d41deab29dc\") " pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" Nov 02 23:07:19.662371 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:19.662300 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfxw6\" (UniqueName: \"kubernetes.io/projected/c5cd8831-584f-47bb-9580-4d41deab29dc-kube-api-access-xfxw6\") pod \"premium-w4-fb6e1d-kserve-64d7869d66-8rzmv\" (UID: \"c5cd8831-584f-47bb-9580-4d41deab29dc\") " pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" Nov 02 23:07:19.695434 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:19.695407 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" Nov 02 23:07:20.078964 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:20.078938 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv"] Nov 02 23:07:20.080502 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:07:20.080473 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5cd8831_584f_47bb_9580_4d41deab29dc.slice/crio-bcedc58f7537d733ad04b57bcb750bf880f8f2ae3ca9b72235107c203cc4b5df WatchSource:0}: Error finding container bcedc58f7537d733ad04b57bcb750bf880f8f2ae3ca9b72235107c203cc4b5df: Status 404 returned error can't find the container with id bcedc58f7537d733ad04b57bcb750bf880f8f2ae3ca9b72235107c203cc4b5df Nov 02 23:07:20.754388 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:20.754349 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" event={"ID":"c5cd8831-584f-47bb-9580-4d41deab29dc","Type":"ContainerStarted","Data":"cf6208e292a38fc8dda1168ff8fe0f6913b3ffd612d5ba88a732ae89136a2ad8"} Nov 02 23:07:20.754807 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:20.754397 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" event={"ID":"c5cd8831-584f-47bb-9580-4d41deab29dc","Type":"ContainerStarted","Data":"bcedc58f7537d733ad04b57bcb750bf880f8f2ae3ca9b72235107c203cc4b5df"} Nov 02 23:07:20.864795 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:20.864732 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" podStartSLOduration=1.864717864 podStartE2EDuration="1.864717864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:07:20.862710442 +0000 UTC m=+889.831242228" watchObservedRunningTime="2026-11-02 23:07:20.864717864 +0000 UTC m=+889.833249648" Nov 02 23:07:21.696003 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:21.695969 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" Nov 02 23:07:29.695856 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:29.695817 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" Nov 02 23:07:29.708766 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:29.708738 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" Nov 02 23:07:29.815306 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:29.815278 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" Nov 02 23:07:31.256679 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:31.256652 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 02 23:07:31.264629 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:31.264604 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 02 23:07:39.883337 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:07:39.883299 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:07:59.455799 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:59.455763 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9"] Nov 02 23:07:59.459789 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:59.459771 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" Nov 02 23:07:59.509261 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:59.509233 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-fb6e1d-openshift-default-dockercfg-5lz5h\"" Nov 02 23:07:59.531706 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:59.531671 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6815fb-2006-4710-a166-db066f8359f8-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9\" (UID: \"5b6815fb-2006-4710-a166-db066f8359f8\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" Nov 02 23:07:59.531866 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:59.531722 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b6815fb-2006-4710-a166-db066f8359f8-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9\" (UID: \"5b6815fb-2006-4710-a166-db066f8359f8\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" Nov 02 23:07:59.531866 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:59.531840 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b6815fb-2006-4710-a166-db066f8359f8-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9\" (UID: \"5b6815fb-2006-4710-a166-db066f8359f8\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" Nov 02 23:07:59.531957 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:59.531870 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6815fb-2006-4710-a166-db066f8359f8-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9\" (UID: \"5b6815fb-2006-4710-a166-db066f8359f8\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" Nov 02 23:07:59.531957 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:59.531928 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b6815fb-2006-4710-a166-db066f8359f8-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9\" (UID: \"5b6815fb-2006-4710-a166-db066f8359f8\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" Nov 02 23:07:59.532027 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:59.532018 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b6815fb-2006-4710-a166-db066f8359f8-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9\" (UID: \"5b6815fb-2006-4710-a166-db066f8359f8\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" Nov 02 23:07:59.532062 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:59.532048 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzbgc\" (UniqueName: \"kubernetes.io/projected/5b6815fb-2006-4710-a166-db066f8359f8-kube-api-access-xzbgc\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9\" (UID: \"5b6815fb-2006-4710-a166-db066f8359f8\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" Nov 02 23:07:59.532096 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:59.532063 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b6815fb-2006-4710-a166-db066f8359f8-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9\" (UID: \"5b6815fb-2006-4710-a166-db066f8359f8\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" Nov 02 23:07:59.532096 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:59.532090 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b6815fb-2006-4710-a166-db066f8359f8-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9\" (UID: \"5b6815fb-2006-4710-a166-db066f8359f8\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" Nov 02 23:07:59.559639 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:59.559597 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9"] Nov 02 23:07:59.633528 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:59.633492 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b6815fb-2006-4710-a166-db066f8359f8-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9\" (UID: \"5b6815fb-2006-4710-a166-db066f8359f8\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" Nov 02 23:07:59.633528 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:59.633529 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6815fb-2006-4710-a166-db066f8359f8-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9\" (UID: \"5b6815fb-2006-4710-a166-db066f8359f8\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" Nov 02 23:07:59.633779 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:59.633548 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b6815fb-2006-4710-a166-db066f8359f8-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9\" (UID: \"5b6815fb-2006-4710-a166-db066f8359f8\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" Nov 02 23:07:59.633779 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:59.633602 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b6815fb-2006-4710-a166-db066f8359f8-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9\" (UID: \"5b6815fb-2006-4710-a166-db066f8359f8\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" Nov 02 23:07:59.633779 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:59.633628 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzbgc\" (UniqueName: \"kubernetes.io/projected/5b6815fb-2006-4710-a166-db066f8359f8-kube-api-access-xzbgc\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9\" (UID: \"5b6815fb-2006-4710-a166-db066f8359f8\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" Nov 02 23:07:59.633779 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:59.633648 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b6815fb-2006-4710-a166-db066f8359f8-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9\" (UID: \"5b6815fb-2006-4710-a166-db066f8359f8\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" Nov 02 23:07:59.633779 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:59.633674 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b6815fb-2006-4710-a166-db066f8359f8-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9\" (UID: \"5b6815fb-2006-4710-a166-db066f8359f8\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" Nov 02 23:07:59.633779 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:59.633741 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6815fb-2006-4710-a166-db066f8359f8-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9\" (UID: \"5b6815fb-2006-4710-a166-db066f8359f8\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" Nov 02 23:07:59.633779 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:59.633769 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b6815fb-2006-4710-a166-db066f8359f8-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9\" (UID: \"5b6815fb-2006-4710-a166-db066f8359f8\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" Nov 02 23:07:59.634167 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:59.634010 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6815fb-2006-4710-a166-db066f8359f8-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9\" (UID: \"5b6815fb-2006-4710-a166-db066f8359f8\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" Nov 02 23:07:59.634167 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:59.634138 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b6815fb-2006-4710-a166-db066f8359f8-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9\" (UID: \"5b6815fb-2006-4710-a166-db066f8359f8\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" Nov 02 23:07:59.634240 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:59.634168 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b6815fb-2006-4710-a166-db066f8359f8-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9\" (UID: \"5b6815fb-2006-4710-a166-db066f8359f8\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" Nov 02 23:07:59.634240 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:59.634217 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6815fb-2006-4710-a166-db066f8359f8-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9\" (UID: \"5b6815fb-2006-4710-a166-db066f8359f8\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" Nov 02 23:07:59.634454 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:59.634423 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b6815fb-2006-4710-a166-db066f8359f8-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9\" (UID: \"5b6815fb-2006-4710-a166-db066f8359f8\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" Nov 02 23:07:59.635918 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:59.635879 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b6815fb-2006-4710-a166-db066f8359f8-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9\" (UID: \"5b6815fb-2006-4710-a166-db066f8359f8\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" Nov 02 23:07:59.636401 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:59.636382 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b6815fb-2006-4710-a166-db066f8359f8-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9\" (UID: \"5b6815fb-2006-4710-a166-db066f8359f8\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" Nov 02 23:07:59.687600 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:59.687561 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b6815fb-2006-4710-a166-db066f8359f8-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9\" (UID: \"5b6815fb-2006-4710-a166-db066f8359f8\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" Nov 02 23:07:59.687780 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:59.687710 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzbgc\" (UniqueName: \"kubernetes.io/projected/5b6815fb-2006-4710-a166-db066f8359f8-kube-api-access-xzbgc\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9\" (UID: \"5b6815fb-2006-4710-a166-db066f8359f8\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" Nov 02 23:07:59.772763 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:07:59.772677 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" Nov 02 23:08:00.189145 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:00.189116 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9"] Nov 02 23:08:00.190239 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:08:00.190211 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b6815fb_2006_4710_a166_db066f8359f8.slice/crio-95b9dd08e556991b54f39d1769549e3c6a5c36eb25b264aa07368853caa4def3 WatchSource:0}: Error finding container 95b9dd08e556991b54f39d1769549e3c6a5c36eb25b264aa07368853caa4def3: Status 404 returned error can't find the container with id 95b9dd08e556991b54f39d1769549e3c6a5c36eb25b264aa07368853caa4def3 Nov 02 23:08:00.192540 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:00.192507 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:08:00.192635 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:00.192588 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:08:00.192698 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:00.192635 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:08:00.976871 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:00.976837 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" event={"ID":"5b6815fb-2006-4710-a166-db066f8359f8","Type":"ContainerStarted","Data":"2911d3e7565a9a9350d7419ecbcd958bf5fc23a82590df7e00f74e91e821e193"} Nov 02 23:08:00.977355 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:00.976928 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" event={"ID":"5b6815fb-2006-4710-a166-db066f8359f8","Type":"ContainerStarted","Data":"95b9dd08e556991b54f39d1769549e3c6a5c36eb25b264aa07368853caa4def3"} Nov 02 23:08:01.189560 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:01.189508 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" podStartSLOduration=2.189495551 podStartE2EDuration="2.189495551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:07:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:08:01.142115538 +0000 UTC m=+930.110647322" watchObservedRunningTime="2026-11-02 23:08:01.189495551 +0000 UTC m=+930.158027335" Nov 02 23:08:01.773731 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:01.773693 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" Nov 02 23:08:01.775615 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:01.775588 5076 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 02 23:08:01.775748 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:01.775646 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" podUID="5b6815fb-2006-4710-a166-db066f8359f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 02 23:08:02.773946 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:02.773881 5076 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 02 23:08:02.774365 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:02.773976 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" podUID="5b6815fb-2006-4710-a166-db066f8359f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 02 23:08:03.773753 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:03.773715 5076 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 02 23:08:03.773954 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:03.773777 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" podUID="5b6815fb-2006-4710-a166-db066f8359f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 02 23:08:04.775549 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:04.775506 5076 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 02 23:08:04.776288 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:04.775624 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" podUID="5b6815fb-2006-4710-a166-db066f8359f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 02 23:08:05.773424 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:05.773385 5076 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 02 23:08:05.773613 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:05.773438 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" podUID="5b6815fb-2006-4710-a166-db066f8359f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 02 23:08:06.878387 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:06.878066 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" Nov 02 23:08:06.879194 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:06.879173 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" Nov 02 23:08:06.880865 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:06.880840 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" Nov 02 23:08:09.319978 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:09.319945 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5f5b4df8c-8kdbn"] Nov 02 23:08:09.324975 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:09.324951 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5f5b4df8c-8kdbn" Nov 02 23:08:09.377490 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:09.377453 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f5b4df8c-8kdbn"] Nov 02 23:08:09.422283 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:09.422251 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-ef36b3c11cbb\"" Nov 02 23:08:09.422478 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:09.422303 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 23:08:09.422478 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:09.422364 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 23:08:09.422478 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:09.422410 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 23:08:09.422478 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:09.422426 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-d5hk8\"" Nov 02 23:08:09.422478 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:09.422462 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-4e90176b6698\"" Nov 02 23:08:09.422703 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:09.422547 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 23:08:09.473334 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:09.473296 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f8b59cd8-c08c-43af-9345-edba8a21e667-routing-overlay-volume\") pod \"payload-processing-external-model-5f5b4df8c-8kdbn\" (UID: \"f8b59cd8-c08c-43af-9345-edba8a21e667\") " pod="models-as-a-service/payload-processing-external-model-5f5b4df8c-8kdbn" Nov 02 23:08:09.473505 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:09.473415 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtr4c\" (UniqueName: \"kubernetes.io/projected/f8b59cd8-c08c-43af-9345-edba8a21e667-kube-api-access-gtr4c\") pod \"payload-processing-external-model-5f5b4df8c-8kdbn\" (UID: \"f8b59cd8-c08c-43af-9345-edba8a21e667\") " pod="models-as-a-service/payload-processing-external-model-5f5b4df8c-8kdbn" Nov 02 23:08:09.473505 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:09.473485 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8b59cd8-c08c-43af-9345-edba8a21e667-plugins-config-volume\") pod \"payload-processing-external-model-5f5b4df8c-8kdbn\" (UID: \"f8b59cd8-c08c-43af-9345-edba8a21e667\") " pod="models-as-a-service/payload-processing-external-model-5f5b4df8c-8kdbn" Nov 02 23:08:09.473619 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:09.473599 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f8b59cd8-c08c-43af-9345-edba8a21e667-provider-credentials\") pod \"payload-processing-external-model-5f5b4df8c-8kdbn\" (UID: \"f8b59cd8-c08c-43af-9345-edba8a21e667\") " pod="models-as-a-service/payload-processing-external-model-5f5b4df8c-8kdbn" Nov 02 23:08:09.574792 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:09.574699 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8b59cd8-c08c-43af-9345-edba8a21e667-plugins-config-volume\") pod \"payload-processing-external-model-5f5b4df8c-8kdbn\" (UID: \"f8b59cd8-c08c-43af-9345-edba8a21e667\") " pod="models-as-a-service/payload-processing-external-model-5f5b4df8c-8kdbn" Nov 02 23:08:09.574993 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:09.574790 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f8b59cd8-c08c-43af-9345-edba8a21e667-provider-credentials\") pod \"payload-processing-external-model-5f5b4df8c-8kdbn\" (UID: \"f8b59cd8-c08c-43af-9345-edba8a21e667\") " pod="models-as-a-service/payload-processing-external-model-5f5b4df8c-8kdbn" Nov 02 23:08:09.574993 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:09.574876 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f8b59cd8-c08c-43af-9345-edba8a21e667-routing-overlay-volume\") pod \"payload-processing-external-model-5f5b4df8c-8kdbn\" (UID: \"f8b59cd8-c08c-43af-9345-edba8a21e667\") " pod="models-as-a-service/payload-processing-external-model-5f5b4df8c-8kdbn" Nov 02 23:08:09.574993 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:09.574967 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtr4c\" (UniqueName: \"kubernetes.io/projected/f8b59cd8-c08c-43af-9345-edba8a21e667-kube-api-access-gtr4c\") pod \"payload-processing-external-model-5f5b4df8c-8kdbn\" (UID: \"f8b59cd8-c08c-43af-9345-edba8a21e667\") " pod="models-as-a-service/payload-processing-external-model-5f5b4df8c-8kdbn" Nov 02 23:08:09.575500 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:09.575472 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f8b59cd8-c08c-43af-9345-edba8a21e667-routing-overlay-volume\") pod \"payload-processing-external-model-5f5b4df8c-8kdbn\" (UID: \"f8b59cd8-c08c-43af-9345-edba8a21e667\") " pod="models-as-a-service/payload-processing-external-model-5f5b4df8c-8kdbn" Nov 02 23:08:09.575730 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:09.575709 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8b59cd8-c08c-43af-9345-edba8a21e667-plugins-config-volume\") pod \"payload-processing-external-model-5f5b4df8c-8kdbn\" (UID: \"f8b59cd8-c08c-43af-9345-edba8a21e667\") " pod="models-as-a-service/payload-processing-external-model-5f5b4df8c-8kdbn" Nov 02 23:08:09.577391 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:09.577370 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f8b59cd8-c08c-43af-9345-edba8a21e667-provider-credentials\") pod \"payload-processing-external-model-5f5b4df8c-8kdbn\" (UID: \"f8b59cd8-c08c-43af-9345-edba8a21e667\") " pod="models-as-a-service/payload-processing-external-model-5f5b4df8c-8kdbn" Nov 02 23:08:09.655114 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:09.655051 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtr4c\" (UniqueName: \"kubernetes.io/projected/f8b59cd8-c08c-43af-9345-edba8a21e667-kube-api-access-gtr4c\") pod \"payload-processing-external-model-5f5b4df8c-8kdbn\" (UID: \"f8b59cd8-c08c-43af-9345-edba8a21e667\") " pod="models-as-a-service/payload-processing-external-model-5f5b4df8c-8kdbn" Nov 02 23:08:09.885207 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:08:09.885111 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:08:09.939420 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:09.939364 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5f5b4df8c-8kdbn" Nov 02 23:08:10.310116 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:10.310087 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f5b4df8c-8kdbn"] Nov 02 23:08:10.312131 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:08:10.312102 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b59cd8_c08c_43af_9345_edba8a21e667.slice/crio-0517efaa9f5b0f1781198ea32c080b8df21846533fe260d12f4d386f954f391d WatchSource:0}: Error finding container 0517efaa9f5b0f1781198ea32c080b8df21846533fe260d12f4d386f954f391d: Status 404 returned error can't find the container with id 0517efaa9f5b0f1781198ea32c080b8df21846533fe260d12f4d386f954f391d Nov 02 23:08:11.064992 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:11.064950 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5f5b4df8c-8kdbn" event={"ID":"f8b59cd8-c08c-43af-9345-edba8a21e667","Type":"ContainerStarted","Data":"a41f06ae95e1c27ba603cd825519f500cb58132c86524d7e261356d05dafddd0"} Nov 02 23:08:11.065537 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:11.065020 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5f5b4df8c-8kdbn" event={"ID":"f8b59cd8-c08c-43af-9345-edba8a21e667","Type":"ContainerStarted","Data":"0517efaa9f5b0f1781198ea32c080b8df21846533fe260d12f4d386f954f391d"} Nov 02 23:08:11.065537 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:11.065144 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5f5b4df8c-8kdbn" Nov 02 23:08:11.177234 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:11.177132 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5f5b4df8c-8kdbn" podStartSLOduration=2.177117502 podStartE2EDuration="2.177117502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:08:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:08:11.173553192 +0000 UTC m=+940.142084984" watchObservedRunningTime="2026-11-02 23:08:11.177117502 +0000 UTC m=+940.145649285" Nov 02 23:08:11.682760 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:11.682720 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb"] Nov 02 23:08:11.686712 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:11.686690 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" Nov 02 23:08:11.713301 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:11.713261 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkpkq\" (UniqueName: \"kubernetes.io/projected/8dd885dc-9e61-4303-a916-f422ab5eb322-kube-api-access-dkpkq\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb\" (UID: \"8dd885dc-9e61-4303-a916-f422ab5eb322\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" Nov 02 23:08:11.713516 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:11.713326 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd885dc-9e61-4303-a916-f422ab5eb322-credential-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb\" (UID: \"8dd885dc-9e61-4303-a916-f422ab5eb322\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" Nov 02 23:08:11.713516 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:11.713391 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dd885dc-9e61-4303-a916-f422ab5eb322-istiod-ca-cert\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb\" (UID: \"8dd885dc-9e61-4303-a916-f422ab5eb322\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" Nov 02 23:08:11.713659 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:11.713523 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dd885dc-9e61-4303-a916-f422ab5eb322-istio-podinfo\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb\" (UID: \"8dd885dc-9e61-4303-a916-f422ab5eb322\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" Nov 02 23:08:11.713659 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:11.713578 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dd885dc-9e61-4303-a916-f422ab5eb322-istio-data\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb\" (UID: \"8dd885dc-9e61-4303-a916-f422ab5eb322\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" Nov 02 23:08:11.713659 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:11.713623 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd885dc-9e61-4303-a916-f422ab5eb322-workload-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb\" (UID: \"8dd885dc-9e61-4303-a916-f422ab5eb322\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" Nov 02 23:08:11.713827 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:11.713672 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dd885dc-9e61-4303-a916-f422ab5eb322-workload-certs\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb\" (UID: \"8dd885dc-9e61-4303-a916-f422ab5eb322\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" Nov 02 23:08:11.713827 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:11.713729 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dd885dc-9e61-4303-a916-f422ab5eb322-istio-envoy\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb\" (UID: \"8dd885dc-9e61-4303-a916-f422ab5eb322\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" Nov 02 23:08:11.713827 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:11.713755 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dd885dc-9e61-4303-a916-f422ab5eb322-istio-token\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb\" (UID: \"8dd885dc-9e61-4303-a916-f422ab5eb322\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" Nov 02 23:08:11.783816 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:11.783767 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-3d8404-openshift-default-dockercfg-kh456\"" Nov 02 23:08:11.786964 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:11.786928 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb"] Nov 02 23:08:11.814734 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:11.814694 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkpkq\" (UniqueName: \"kubernetes.io/projected/8dd885dc-9e61-4303-a916-f422ab5eb322-kube-api-access-dkpkq\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb\" (UID: \"8dd885dc-9e61-4303-a916-f422ab5eb322\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" Nov 02 23:08:11.814981 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:11.814757 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd885dc-9e61-4303-a916-f422ab5eb322-credential-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb\" (UID: \"8dd885dc-9e61-4303-a916-f422ab5eb322\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" Nov 02 23:08:11.814981 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:11.814786 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dd885dc-9e61-4303-a916-f422ab5eb322-istiod-ca-cert\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb\" (UID: \"8dd885dc-9e61-4303-a916-f422ab5eb322\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" Nov 02 23:08:11.814981 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:11.814825 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dd885dc-9e61-4303-a916-f422ab5eb322-istio-podinfo\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb\" (UID: \"8dd885dc-9e61-4303-a916-f422ab5eb322\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" Nov 02 23:08:11.814981 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:11.814853 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dd885dc-9e61-4303-a916-f422ab5eb322-istio-data\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb\" (UID: \"8dd885dc-9e61-4303-a916-f422ab5eb322\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" Nov 02 23:08:11.814981 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:11.814921 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd885dc-9e61-4303-a916-f422ab5eb322-workload-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb\" (UID: \"8dd885dc-9e61-4303-a916-f422ab5eb322\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" Nov 02 23:08:11.814981 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:11.814954 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dd885dc-9e61-4303-a916-f422ab5eb322-workload-certs\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb\" (UID: \"8dd885dc-9e61-4303-a916-f422ab5eb322\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" Nov 02 23:08:11.815354 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:11.815007 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dd885dc-9e61-4303-a916-f422ab5eb322-istio-envoy\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb\" (UID: \"8dd885dc-9e61-4303-a916-f422ab5eb322\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" Nov 02 23:08:11.815354 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:11.815030 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dd885dc-9e61-4303-a916-f422ab5eb322-istio-token\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb\" (UID: \"8dd885dc-9e61-4303-a916-f422ab5eb322\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" Nov 02 23:08:11.816198 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:11.816172 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dd885dc-9e61-4303-a916-f422ab5eb322-istio-data\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb\" (UID: \"8dd885dc-9e61-4303-a916-f422ab5eb322\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" Nov 02 23:08:11.816717 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:11.816696 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd885dc-9e61-4303-a916-f422ab5eb322-workload-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb\" (UID: \"8dd885dc-9e61-4303-a916-f422ab5eb322\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" Nov 02 23:08:11.817085 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:11.817051 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dd885dc-9e61-4303-a916-f422ab5eb322-istiod-ca-cert\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb\" (UID: \"8dd885dc-9e61-4303-a916-f422ab5eb322\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" Nov 02 23:08:11.817524 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:11.817496 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd885dc-9e61-4303-a916-f422ab5eb322-credential-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb\" (UID: \"8dd885dc-9e61-4303-a916-f422ab5eb322\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" Nov 02 23:08:11.817808 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:11.817787 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dd885dc-9e61-4303-a916-f422ab5eb322-workload-certs\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb\" (UID: \"8dd885dc-9e61-4303-a916-f422ab5eb322\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" Nov 02 23:08:11.819726 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:11.819701 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dd885dc-9e61-4303-a916-f422ab5eb322-istio-podinfo\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb\" (UID: \"8dd885dc-9e61-4303-a916-f422ab5eb322\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" Nov 02 23:08:11.822329 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:11.822304 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dd885dc-9e61-4303-a916-f422ab5eb322-istio-envoy\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb\" (UID: \"8dd885dc-9e61-4303-a916-f422ab5eb322\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" Nov 02 23:08:11.869366 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:11.869336 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dd885dc-9e61-4303-a916-f422ab5eb322-istio-token\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb\" (UID: \"8dd885dc-9e61-4303-a916-f422ab5eb322\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" Nov 02 23:08:11.869522 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:11.869499 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkpkq\" (UniqueName: \"kubernetes.io/projected/8dd885dc-9e61-4303-a916-f422ab5eb322-kube-api-access-dkpkq\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb\" (UID: \"8dd885dc-9e61-4303-a916-f422ab5eb322\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" Nov 02 23:08:12.000880 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:12.000840 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" Nov 02 23:08:12.415148 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:12.415118 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb"] Nov 02 23:08:12.416449 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:08:12.416419 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dd885dc_9e61_4303_a916_f422ab5eb322.slice/crio-5f9d64653bf1b476650b136748f1ced0d88a4865be42ed1f15455a2be7bde197 WatchSource:0}: Error finding container 5f9d64653bf1b476650b136748f1ced0d88a4865be42ed1f15455a2be7bde197: Status 404 returned error can't find the container with id 5f9d64653bf1b476650b136748f1ced0d88a4865be42ed1f15455a2be7bde197 Nov 02 23:08:12.418705 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:12.418677 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:08:12.418772 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:12.418739 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:08:12.418805 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:12.418783 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:08:13.077219 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:13.077181 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" event={"ID":"8dd885dc-9e61-4303-a916-f422ab5eb322","Type":"ContainerStarted","Data":"5d1bfd0540626aa083b146cfb5334b059482b8fca8d4e6531a28d9fa58b01412"} Nov 02 23:08:13.077409 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:13.077233 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" event={"ID":"8dd885dc-9e61-4303-a916-f422ab5eb322","Type":"ContainerStarted","Data":"5f9d64653bf1b476650b136748f1ced0d88a4865be42ed1f15455a2be7bde197"} Nov 02 23:08:13.261138 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:13.261076 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" podStartSLOduration=2.261063101 podStartE2EDuration="2.261063101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:08:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:08:13.26054006 +0000 UTC m=+942.229071852" watchObservedRunningTime="2026-11-02 23:08:13.261063101 +0000 UTC m=+942.229594885" Nov 02 23:08:14.001361 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:14.001323 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" Nov 02 23:08:14.003285 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:14.003260 5076 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 02 23:08:14.003427 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:14.003313 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" podUID="8dd885dc-9e61-4303-a916-f422ab5eb322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 02 23:08:15.002156 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:15.002124 5076 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 02 23:08:15.002577 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:15.002193 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" podUID="8dd885dc-9e61-4303-a916-f422ab5eb322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 02 23:08:16.001726 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:16.001685 5076 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 02 23:08:16.001929 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:16.001743 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" podUID="8dd885dc-9e61-4303-a916-f422ab5eb322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 02 23:08:17.002179 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:17.002145 5076 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 02 23:08:17.002568 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:17.002197 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" podUID="8dd885dc-9e61-4303-a916-f422ab5eb322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 02 23:08:18.001402 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:18.001367 5076 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 02 23:08:18.001585 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:18.001418 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" podUID="8dd885dc-9e61-4303-a916-f422ab5eb322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 02 23:08:19.036002 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:19.035968 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" Nov 02 23:08:19.036504 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:19.036324 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" Nov 02 23:08:19.036999 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:19.036978 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" Nov 02 23:08:22.076187 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:22.075321 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5f5b4df8c-8kdbn" Nov 02 23:08:30.485300 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:30.485259 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h"] Nov 02 23:08:30.489508 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:30.489483 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" Nov 02 23:08:30.549736 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:30.549705 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l658\" (UniqueName: \"kubernetes.io/projected/db841b5b-fade-48e4-9515-ef923b888f42-kube-api-access-5l658\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vfl6h\" (UID: \"db841b5b-fade-48e4-9515-ef923b888f42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" Nov 02 23:08:30.549943 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:30.549742 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db841b5b-fade-48e4-9515-ef923b888f42-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vfl6h\" (UID: \"db841b5b-fade-48e4-9515-ef923b888f42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" Nov 02 23:08:30.549943 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:30.549764 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db841b5b-fade-48e4-9515-ef923b888f42-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vfl6h\" (UID: \"db841b5b-fade-48e4-9515-ef923b888f42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" Nov 02 23:08:30.549943 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:30.549783 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db841b5b-fade-48e4-9515-ef923b888f42-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vfl6h\" (UID: \"db841b5b-fade-48e4-9515-ef923b888f42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" Nov 02 23:08:30.549943 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:30.549884 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db841b5b-fade-48e4-9515-ef923b888f42-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vfl6h\" (UID: \"db841b5b-fade-48e4-9515-ef923b888f42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" Nov 02 23:08:30.550090 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:30.549952 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db841b5b-fade-48e4-9515-ef923b888f42-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vfl6h\" (UID: \"db841b5b-fade-48e4-9515-ef923b888f42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" Nov 02 23:08:30.550090 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:30.549993 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db841b5b-fade-48e4-9515-ef923b888f42-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vfl6h\" (UID: \"db841b5b-fade-48e4-9515-ef923b888f42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" Nov 02 23:08:30.550090 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:30.550048 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db841b5b-fade-48e4-9515-ef923b888f42-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vfl6h\" (UID: \"db841b5b-fade-48e4-9515-ef923b888f42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" Nov 02 23:08:30.550178 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:30.550096 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db841b5b-fade-48e4-9515-ef923b888f42-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vfl6h\" (UID: \"db841b5b-fade-48e4-9515-ef923b888f42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" Nov 02 23:08:30.590942 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:30.590908 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h"] Nov 02 23:08:30.651239 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:30.651207 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db841b5b-fade-48e4-9515-ef923b888f42-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vfl6h\" (UID: \"db841b5b-fade-48e4-9515-ef923b888f42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" Nov 02 23:08:30.651408 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:30.651247 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db841b5b-fade-48e4-9515-ef923b888f42-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vfl6h\" (UID: \"db841b5b-fade-48e4-9515-ef923b888f42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" Nov 02 23:08:30.651408 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:30.651278 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db841b5b-fade-48e4-9515-ef923b888f42-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vfl6h\" (UID: \"db841b5b-fade-48e4-9515-ef923b888f42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" Nov 02 23:08:30.651408 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:30.651324 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db841b5b-fade-48e4-9515-ef923b888f42-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vfl6h\" (UID: \"db841b5b-fade-48e4-9515-ef923b888f42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" Nov 02 23:08:30.651598 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:30.651414 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db841b5b-fade-48e4-9515-ef923b888f42-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vfl6h\" (UID: \"db841b5b-fade-48e4-9515-ef923b888f42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" Nov 02 23:08:30.651598 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:30.651443 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db841b5b-fade-48e4-9515-ef923b888f42-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vfl6h\" (UID: \"db841b5b-fade-48e4-9515-ef923b888f42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" Nov 02 23:08:30.651598 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:30.651491 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db841b5b-fade-48e4-9515-ef923b888f42-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vfl6h\" (UID: \"db841b5b-fade-48e4-9515-ef923b888f42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" Nov 02 23:08:30.651598 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:30.651549 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db841b5b-fade-48e4-9515-ef923b888f42-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vfl6h\" (UID: \"db841b5b-fade-48e4-9515-ef923b888f42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" Nov 02 23:08:30.651814 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:30.651606 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l658\" (UniqueName: \"kubernetes.io/projected/db841b5b-fade-48e4-9515-ef923b888f42-kube-api-access-5l658\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vfl6h\" (UID: \"db841b5b-fade-48e4-9515-ef923b888f42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" Nov 02 23:08:30.652173 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:30.652148 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db841b5b-fade-48e4-9515-ef923b888f42-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vfl6h\" (UID: \"db841b5b-fade-48e4-9515-ef923b888f42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" Nov 02 23:08:30.652312 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:30.652282 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db841b5b-fade-48e4-9515-ef923b888f42-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vfl6h\" (UID: \"db841b5b-fade-48e4-9515-ef923b888f42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" Nov 02 23:08:30.652473 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:30.652449 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db841b5b-fade-48e4-9515-ef923b888f42-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vfl6h\" (UID: \"db841b5b-fade-48e4-9515-ef923b888f42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" Nov 02 23:08:30.652548 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:30.652470 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db841b5b-fade-48e4-9515-ef923b888f42-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vfl6h\" (UID: \"db841b5b-fade-48e4-9515-ef923b888f42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" Nov 02 23:08:30.652612 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:30.652573 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db841b5b-fade-48e4-9515-ef923b888f42-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vfl6h\" (UID: \"db841b5b-fade-48e4-9515-ef923b888f42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" Nov 02 23:08:30.653882 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:30.653861 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db841b5b-fade-48e4-9515-ef923b888f42-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vfl6h\" (UID: \"db841b5b-fade-48e4-9515-ef923b888f42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" Nov 02 23:08:30.654321 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:30.654305 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db841b5b-fade-48e4-9515-ef923b888f42-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vfl6h\" (UID: \"db841b5b-fade-48e4-9515-ef923b888f42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" Nov 02 23:08:30.706537 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:30.706497 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db841b5b-fade-48e4-9515-ef923b888f42-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vfl6h\" (UID: \"db841b5b-fade-48e4-9515-ef923b888f42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" Nov 02 23:08:30.706690 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:30.706668 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l658\" (UniqueName: \"kubernetes.io/projected/db841b5b-fade-48e4-9515-ef923b888f42-kube-api-access-5l658\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vfl6h\" (UID: \"db841b5b-fade-48e4-9515-ef923b888f42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" Nov 02 23:08:30.803193 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:30.803106 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" Nov 02 23:08:31.221646 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:31.221622 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h"] Nov 02 23:08:31.223140 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:08:31.223110 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb841b5b_fade_48e4_9515_ef923b888f42.slice/crio-2b0c984aaaff34667ef6334ae47576303eda8c9ba7ebaa2b1cf6375bbe5215b0 WatchSource:0}: Error finding container 2b0c984aaaff34667ef6334ae47576303eda8c9ba7ebaa2b1cf6375bbe5215b0: Status 404 returned error can't find the container with id 2b0c984aaaff34667ef6334ae47576303eda8c9ba7ebaa2b1cf6375bbe5215b0 Nov 02 23:08:31.225388 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:31.225358 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:08:31.225490 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:31.225434 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:08:31.225490 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:31.225476 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:08:32.195643 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:32.195600 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" event={"ID":"db841b5b-fade-48e4-9515-ef923b888f42","Type":"ContainerStarted","Data":"50bebcfe439db99de3dd447638369c5a25e6b60507a29e399e952c04c69fadc9"} Nov 02 23:08:32.196103 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:32.195691 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" event={"ID":"db841b5b-fade-48e4-9515-ef923b888f42","Type":"ContainerStarted","Data":"2b0c984aaaff34667ef6334ae47576303eda8c9ba7ebaa2b1cf6375bbe5215b0"} Nov 02 23:08:32.357925 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:32.357853 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" podStartSLOduration=2.3578396169999998 podStartE2EDuration="2.357839617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:08:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:08:32.355962306 +0000 UTC m=+961.324494088" watchObservedRunningTime="2026-11-02 23:08:32.357839617 +0000 UTC m=+961.326371401" Nov 02 23:08:32.803653 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:32.803625 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" Nov 02 23:08:32.805425 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:32.805400 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 02 23:08:32.805543 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:32.805454 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" podUID="db841b5b-fade-48e4-9515-ef923b888f42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 02 23:08:33.804373 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:33.804331 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 02 23:08:33.804782 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:33.804397 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" podUID="db841b5b-fade-48e4-9515-ef923b888f42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 02 23:08:34.804484 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:34.804450 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 02 23:08:34.804930 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:34.804500 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" podUID="db841b5b-fade-48e4-9515-ef923b888f42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 02 23:08:35.804193 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:35.804152 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 02 23:08:35.804421 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:35.804213 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" podUID="db841b5b-fade-48e4-9515-ef923b888f42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 02 23:08:36.804096 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:36.804060 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 02 23:08:36.804638 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:36.804118 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" podUID="db841b5b-fade-48e4-9515-ef923b888f42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 02 23:08:37.803666 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:37.803629 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 02 23:08:37.803869 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:37.803697 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" podUID="db841b5b-fade-48e4-9515-ef923b888f42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 02 23:08:38.804070 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:38.804030 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 02 23:08:38.804681 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:38.804642 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" podUID="db841b5b-fade-48e4-9515-ef923b888f42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 02 23:08:39.978959 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:08:39.978916 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:08:40.461516 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:40.459212 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" Nov 02 23:08:40.461516 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:40.460578 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" Nov 02 23:08:40.473022 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:40.472159 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" Nov 02 23:08:45.601973 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:45.601935 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn"] Nov 02 23:08:45.607220 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:45.607201 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" Nov 02 23:08:45.638053 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:45.638021 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a67b3447-b8b2-4485-bacf-0d39a1352feb-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ccjjn\" (UID: \"a67b3447-b8b2-4485-bacf-0d39a1352feb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" Nov 02 23:08:45.638202 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:45.638069 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a67b3447-b8b2-4485-bacf-0d39a1352feb-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ccjjn\" (UID: \"a67b3447-b8b2-4485-bacf-0d39a1352feb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" Nov 02 23:08:45.638202 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:45.638090 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a67b3447-b8b2-4485-bacf-0d39a1352feb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ccjjn\" (UID: \"a67b3447-b8b2-4485-bacf-0d39a1352feb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" Nov 02 23:08:45.638282 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:45.638231 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67b3447-b8b2-4485-bacf-0d39a1352feb-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ccjjn\" (UID: \"a67b3447-b8b2-4485-bacf-0d39a1352feb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" Nov 02 23:08:45.638321 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:45.638280 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a67b3447-b8b2-4485-bacf-0d39a1352feb-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ccjjn\" (UID: \"a67b3447-b8b2-4485-bacf-0d39a1352feb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" Nov 02 23:08:45.638391 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:45.638376 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a67b3447-b8b2-4485-bacf-0d39a1352feb-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ccjjn\" (UID: \"a67b3447-b8b2-4485-bacf-0d39a1352feb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" Nov 02 23:08:45.638426 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:45.638404 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a67b3447-b8b2-4485-bacf-0d39a1352feb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ccjjn\" (UID: \"a67b3447-b8b2-4485-bacf-0d39a1352feb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" Nov 02 23:08:45.638526 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:45.638512 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67b3447-b8b2-4485-bacf-0d39a1352feb-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ccjjn\" (UID: \"a67b3447-b8b2-4485-bacf-0d39a1352feb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" Nov 02 23:08:45.638560 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:45.638534 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8wxz\" (UniqueName: \"kubernetes.io/projected/a67b3447-b8b2-4485-bacf-0d39a1352feb-kube-api-access-n8wxz\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ccjjn\" (UID: \"a67b3447-b8b2-4485-bacf-0d39a1352feb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" Nov 02 23:08:45.711285 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:45.711247 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn"] Nov 02 23:08:45.739435 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:45.739405 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67b3447-b8b2-4485-bacf-0d39a1352feb-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ccjjn\" (UID: \"a67b3447-b8b2-4485-bacf-0d39a1352feb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" Nov 02 23:08:45.739597 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:45.739439 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8wxz\" (UniqueName: \"kubernetes.io/projected/a67b3447-b8b2-4485-bacf-0d39a1352feb-kube-api-access-n8wxz\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ccjjn\" (UID: \"a67b3447-b8b2-4485-bacf-0d39a1352feb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" Nov 02 23:08:45.739597 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:45.739464 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a67b3447-b8b2-4485-bacf-0d39a1352feb-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ccjjn\" (UID: \"a67b3447-b8b2-4485-bacf-0d39a1352feb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" Nov 02 23:08:45.739716 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:45.739636 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a67b3447-b8b2-4485-bacf-0d39a1352feb-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ccjjn\" (UID: \"a67b3447-b8b2-4485-bacf-0d39a1352feb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" Nov 02 23:08:45.739716 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:45.739690 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a67b3447-b8b2-4485-bacf-0d39a1352feb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ccjjn\" (UID: \"a67b3447-b8b2-4485-bacf-0d39a1352feb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" Nov 02 23:08:45.739821 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:45.739764 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67b3447-b8b2-4485-bacf-0d39a1352feb-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ccjjn\" (UID: \"a67b3447-b8b2-4485-bacf-0d39a1352feb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" Nov 02 23:08:45.745510 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:45.740224 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a67b3447-b8b2-4485-bacf-0d39a1352feb-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ccjjn\" (UID: \"a67b3447-b8b2-4485-bacf-0d39a1352feb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" Nov 02 23:08:45.745510 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:45.740326 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a67b3447-b8b2-4485-bacf-0d39a1352feb-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ccjjn\" (UID: \"a67b3447-b8b2-4485-bacf-0d39a1352feb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" Nov 02 23:08:45.745510 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:45.740348 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67b3447-b8b2-4485-bacf-0d39a1352feb-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ccjjn\" (UID: \"a67b3447-b8b2-4485-bacf-0d39a1352feb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" Nov 02 23:08:45.745510 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:45.740375 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a67b3447-b8b2-4485-bacf-0d39a1352feb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ccjjn\" (UID: \"a67b3447-b8b2-4485-bacf-0d39a1352feb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" Nov 02 23:08:45.745510 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:45.740642 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a67b3447-b8b2-4485-bacf-0d39a1352feb-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ccjjn\" (UID: \"a67b3447-b8b2-4485-bacf-0d39a1352feb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" Nov 02 23:08:45.745510 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:45.740860 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a67b3447-b8b2-4485-bacf-0d39a1352feb-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ccjjn\" (UID: \"a67b3447-b8b2-4485-bacf-0d39a1352feb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" Nov 02 23:08:45.745510 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:45.741324 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a67b3447-b8b2-4485-bacf-0d39a1352feb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ccjjn\" (UID: \"a67b3447-b8b2-4485-bacf-0d39a1352feb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" Nov 02 23:08:45.745510 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:45.742871 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a67b3447-b8b2-4485-bacf-0d39a1352feb-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ccjjn\" (UID: \"a67b3447-b8b2-4485-bacf-0d39a1352feb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" Nov 02 23:08:45.745510 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:45.743139 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a67b3447-b8b2-4485-bacf-0d39a1352feb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ccjjn\" (UID: \"a67b3447-b8b2-4485-bacf-0d39a1352feb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" Nov 02 23:08:45.752464 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:45.752433 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67b3447-b8b2-4485-bacf-0d39a1352feb-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ccjjn\" (UID: \"a67b3447-b8b2-4485-bacf-0d39a1352feb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" Nov 02 23:08:45.793749 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:45.793711 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a67b3447-b8b2-4485-bacf-0d39a1352feb-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ccjjn\" (UID: \"a67b3447-b8b2-4485-bacf-0d39a1352feb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" Nov 02 23:08:45.793875 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:45.793822 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8wxz\" (UniqueName: \"kubernetes.io/projected/a67b3447-b8b2-4485-bacf-0d39a1352feb-kube-api-access-n8wxz\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ccjjn\" (UID: \"a67b3447-b8b2-4485-bacf-0d39a1352feb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" Nov 02 23:08:45.921402 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:45.921315 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" Nov 02 23:08:46.339205 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:46.339177 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn"] Nov 02 23:08:46.340524 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:08:46.340499 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda67b3447_b8b2_4485_bacf_0d39a1352feb.slice/crio-a21c1fa594f6f1311a629027155e98523c521abfa96b5170f4c1b12f7fca9afa WatchSource:0}: Error finding container a21c1fa594f6f1311a629027155e98523c521abfa96b5170f4c1b12f7fca9afa: Status 404 returned error can't find the container with id a21c1fa594f6f1311a629027155e98523c521abfa96b5170f4c1b12f7fca9afa Nov 02 23:08:46.342867 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:46.342832 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:08:46.342985 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:46.342928 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:08:46.342985 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:46.342975 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:08:47.340050 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:47.340010 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" event={"ID":"a67b3447-b8b2-4485-bacf-0d39a1352feb","Type":"ContainerStarted","Data":"6a8adf24919c12f4141a053259df0816b40eebbc8bfb7bdea94a8646b5711b41"} Nov 02 23:08:47.340481 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:47.340076 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" event={"ID":"a67b3447-b8b2-4485-bacf-0d39a1352feb","Type":"ContainerStarted","Data":"a21c1fa594f6f1311a629027155e98523c521abfa96b5170f4c1b12f7fca9afa"} Nov 02 23:08:47.926154 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:47.926066 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" Nov 02 23:08:47.934290 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:47.934254 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 02 23:08:47.934481 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:47.934318 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" podUID="a67b3447-b8b2-4485-bacf-0d39a1352feb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 02 23:08:48.927720 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:48.927272 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 02 23:08:48.927720 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:48.927333 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" podUID="a67b3447-b8b2-4485-bacf-0d39a1352feb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 02 23:08:49.929339 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:49.929302 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 02 23:08:49.930000 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:49.929963 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" podUID="a67b3447-b8b2-4485-bacf-0d39a1352feb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 02 23:08:50.926915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:50.925319 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 02 23:08:50.926915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:50.925383 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" podUID="a67b3447-b8b2-4485-bacf-0d39a1352feb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 02 23:08:51.934918 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:51.933013 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 02 23:08:51.934918 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:51.933080 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" podUID="a67b3447-b8b2-4485-bacf-0d39a1352feb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 02 23:08:52.922028 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:52.921980 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 02 23:08:52.922222 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:52.922038 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" podUID="a67b3447-b8b2-4485-bacf-0d39a1352feb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 02 23:08:53.922164 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:53.922130 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 02 23:08:53.922568 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:53.922184 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" podUID="a67b3447-b8b2-4485-bacf-0d39a1352feb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 02 23:08:55.213518 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:55.213485 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" Nov 02 23:08:55.213984 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:55.213940 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" Nov 02 23:08:55.216342 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:55.216316 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" Nov 02 23:08:55.374014 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:08:55.373956 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" podStartSLOduration=10.373941317 podStartE2EDuration="10.373941317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:08:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:08:47.547921035 +0000 UTC m=+976.516452809" watchObservedRunningTime="2026-11-02 23:08:55.373941317 +0000 UTC m=+984.342473098" Nov 02 23:09:09.885229 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:09:09.885184 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:09:25.982924 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:25.982877 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls"] Nov 02 23:09:25.986930 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:25.986912 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" Nov 02 23:09:26.036656 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:26.036623 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-ba1918-openshift-default-dockercfg-k6bc8\"" Nov 02 23:09:26.087431 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:26.087399 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls"] Nov 02 23:09:26.136938 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:26.136882 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls\" (UID: \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" Nov 02 23:09:26.137105 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:26.136949 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls\" (UID: \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" Nov 02 23:09:26.137105 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:26.137025 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls\" (UID: \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" Nov 02 23:09:26.137105 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:26.137066 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls\" (UID: \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" Nov 02 23:09:26.137211 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:26.137158 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls\" (UID: \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" Nov 02 23:09:26.137211 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:26.137185 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbh8l\" (UniqueName: \"kubernetes.io/projected/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-kube-api-access-xbh8l\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls\" (UID: \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" Nov 02 23:09:26.137292 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:26.137280 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls\" (UID: \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" Nov 02 23:09:26.137328 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:26.137301 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls\" (UID: \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" Nov 02 23:09:26.137445 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:26.137431 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls\" (UID: \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" Nov 02 23:09:26.238661 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:26.238585 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls\" (UID: \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" Nov 02 23:09:26.238661 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:26.238622 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls\" (UID: \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" Nov 02 23:09:26.238851 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:26.238705 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls\" (UID: \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" Nov 02 23:09:26.238851 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:26.238726 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls\" (UID: \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" Nov 02 23:09:26.238851 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:26.238761 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls\" (UID: \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" Nov 02 23:09:26.238851 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:26.238793 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls\" (UID: \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" Nov 02 23:09:26.238851 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:26.238831 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls\" (UID: \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" Nov 02 23:09:26.239136 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:26.238883 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls\" (UID: \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" Nov 02 23:09:26.239136 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:26.238928 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbh8l\" (UniqueName: \"kubernetes.io/projected/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-kube-api-access-xbh8l\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls\" (UID: \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" Nov 02 23:09:26.239319 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:26.239211 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls\" (UID: \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" Nov 02 23:09:26.239319 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:26.239243 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls\" (UID: \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" Nov 02 23:09:26.239418 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:26.239346 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls\" (UID: \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" Nov 02 23:09:26.239496 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:26.239475 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls\" (UID: \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" Nov 02 23:09:26.239496 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:26.239488 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls\" (UID: \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" Nov 02 23:09:26.240990 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:26.240970 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls\" (UID: \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" Nov 02 23:09:26.241354 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:26.241332 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls\" (UID: \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" Nov 02 23:09:26.293088 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:26.293043 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbh8l\" (UniqueName: \"kubernetes.io/projected/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-kube-api-access-xbh8l\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls\" (UID: \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" Nov 02 23:09:26.293855 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:26.293826 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls\" (UID: \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" Nov 02 23:09:26.300676 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:26.300654 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" Nov 02 23:09:26.717701 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:26.717671 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls"] Nov 02 23:09:26.719081 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:09:26.719059 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c2d7a91_9af2_40b7_a010_1a19260cf6f9.slice/crio-8ddf5012b64f1baa714f20ef97822bb926a78fb2da35a08c9fc222fe277a305e WatchSource:0}: Error finding container 8ddf5012b64f1baa714f20ef97822bb926a78fb2da35a08c9fc222fe277a305e: Status 404 returned error can't find the container with id 8ddf5012b64f1baa714f20ef97822bb926a78fb2da35a08c9fc222fe277a305e Nov 02 23:09:26.721553 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:26.721521 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:09:26.721632 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:26.721605 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:09:26.721674 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:26.721639 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:09:27.600933 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:27.600884 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" event={"ID":"0c2d7a91-9af2-40b7-a010-1a19260cf6f9","Type":"ContainerStarted","Data":"12ea9b1a63208befc24d12c9fa7be2120eaefb2fcaece2475a13f4a73265eed7"} Nov 02 23:09:27.601329 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:27.600952 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" event={"ID":"0c2d7a91-9af2-40b7-a010-1a19260cf6f9","Type":"ContainerStarted","Data":"8ddf5012b64f1baa714f20ef97822bb926a78fb2da35a08c9fc222fe277a305e"} Nov 02 23:09:27.765157 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:27.765107 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" podStartSLOduration=2.765093847 podStartE2EDuration="2.765093847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:09:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:09:27.762291619 +0000 UTC m=+1016.730823402" watchObservedRunningTime="2026-11-02 23:09:27.765093847 +0000 UTC m=+1016.733625625" Nov 02 23:09:28.301778 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:28.301719 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" Nov 02 23:09:28.303657 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:28.303617 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 02 23:09:28.303797 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:28.303671 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" podUID="0c2d7a91-9af2-40b7-a010-1a19260cf6f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 02 23:09:29.301470 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:29.301433 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 02 23:09:29.301966 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:29.301497 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" podUID="0c2d7a91-9af2-40b7-a010-1a19260cf6f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 02 23:09:30.301998 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:30.301965 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 02 23:09:30.302397 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:30.302016 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" podUID="0c2d7a91-9af2-40b7-a010-1a19260cf6f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 02 23:09:31.301055 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:31.301021 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 02 23:09:31.301234 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:31.301103 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" podUID="0c2d7a91-9af2-40b7-a010-1a19260cf6f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 02 23:09:32.301133 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:32.301099 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 02 23:09:32.301529 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:32.301150 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" podUID="0c2d7a91-9af2-40b7-a010-1a19260cf6f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 02 23:09:33.301358 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:33.301318 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 02 23:09:33.301766 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:33.301372 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" podUID="0c2d7a91-9af2-40b7-a010-1a19260cf6f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 02 23:09:34.306041 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:34.306012 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" Nov 02 23:09:34.306412 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:34.306276 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" Nov 02 23:09:34.306985 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:34.306969 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" Nov 02 23:09:39.883502 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:09:39.883463 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:09:49.252134 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:09:49.252087 5076 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ef36b3c11cbb: secret "agc-cred-ef36b3c11cbb" not found Nov 02 23:09:49.252134 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:09:49.252119 5076 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5f5b4df8c-8kdbn: secret "agc-cred-ef36b3c11cbb" not found Nov 02 23:09:49.254656 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:09:49.252178 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8b59cd8-c08c-43af-9345-edba8a21e667-provider-credentials podName:f8b59cd8-c08c-43af-9345-edba8a21e667 nodeName:}" failed. No retries permitted until 2026-11-02 23:09:49.752162949 +0000 UTC m=+1038.720694734 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f8b59cd8-c08c-43af-9345-edba8a21e667-provider-credentials") pod "payload-processing-external-model-5f5b4df8c-8kdbn" (UID: "f8b59cd8-c08c-43af-9345-edba8a21e667") : secret "agc-cred-ef36b3c11cbb" not found Nov 02 23:09:49.757339 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:09:49.757303 5076 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ef36b3c11cbb: secret "agc-cred-ef36b3c11cbb" not found Nov 02 23:09:49.757339 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:09:49.757325 5076 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5f5b4df8c-8kdbn: secret "agc-cred-ef36b3c11cbb" not found Nov 02 23:09:49.757553 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:09:49.757381 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8b59cd8-c08c-43af-9345-edba8a21e667-provider-credentials podName:f8b59cd8-c08c-43af-9345-edba8a21e667 nodeName:}" failed. No retries permitted until 2026-11-02 23:09:50.757367179 +0000 UTC m=+1039.725898954 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f8b59cd8-c08c-43af-9345-edba8a21e667-provider-credentials") pod "payload-processing-external-model-5f5b4df8c-8kdbn" (UID: "f8b59cd8-c08c-43af-9345-edba8a21e667") : secret "agc-cred-ef36b3c11cbb" not found Nov 02 23:09:50.780569 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:09:50.780539 5076 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ef36b3c11cbb: secret "agc-cred-ef36b3c11cbb" not found Nov 02 23:09:50.780569 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:09:50.780563 5076 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5f5b4df8c-8kdbn: secret "agc-cred-ef36b3c11cbb" not found Nov 02 23:09:50.781086 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:09:50.780624 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8b59cd8-c08c-43af-9345-edba8a21e667-provider-credentials podName:f8b59cd8-c08c-43af-9345-edba8a21e667 nodeName:}" failed. No retries permitted until 2026-11-02 23:09:52.780610694 +0000 UTC m=+1041.749142456 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f8b59cd8-c08c-43af-9345-edba8a21e667-provider-credentials") pod "payload-processing-external-model-5f5b4df8c-8kdbn" (UID: "f8b59cd8-c08c-43af-9345-edba8a21e667") : secret "agc-cred-ef36b3c11cbb" not found Nov 02 23:09:52.804454 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:09:52.804427 5076 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ef36b3c11cbb: secret "agc-cred-ef36b3c11cbb" not found Nov 02 23:09:52.804454 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:09:52.804450 5076 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5f5b4df8c-8kdbn: secret "agc-cred-ef36b3c11cbb" not found Nov 02 23:09:52.804851 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:09:52.804504 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8b59cd8-c08c-43af-9345-edba8a21e667-provider-credentials podName:f8b59cd8-c08c-43af-9345-edba8a21e667 nodeName:}" failed. No retries permitted until 2026-11-02 23:09:56.804491363 +0000 UTC m=+1045.773023138 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f8b59cd8-c08c-43af-9345-edba8a21e667-provider-credentials") pod "payload-processing-external-model-5f5b4df8c-8kdbn" (UID: "f8b59cd8-c08c-43af-9345-edba8a21e667") : secret "agc-cred-ef36b3c11cbb" not found Nov 02 23:09:56.855647 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:09:56.855605 5076 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ef36b3c11cbb: secret "agc-cred-ef36b3c11cbb" not found Nov 02 23:09:56.855647 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:09:56.855642 5076 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5f5b4df8c-8kdbn: secret "agc-cred-ef36b3c11cbb" not found Nov 02 23:09:56.856130 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:09:56.855701 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8b59cd8-c08c-43af-9345-edba8a21e667-provider-credentials podName:f8b59cd8-c08c-43af-9345-edba8a21e667 nodeName:}" failed. No retries permitted until 2026-11-02 23:10:04.855687558 +0000 UTC m=+1053.824219333 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f8b59cd8-c08c-43af-9345-edba8a21e667-provider-credentials") pod "payload-processing-external-model-5f5b4df8c-8kdbn" (UID: "f8b59cd8-c08c-43af-9345-edba8a21e667") : secret "agc-cred-ef36b3c11cbb" not found Nov 02 23:09:59.315304 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:59.315267 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-88d4f7ff9-xsrng"] Nov 02 23:09:59.319451 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:59.319429 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-88d4f7ff9-xsrng" Nov 02 23:09:59.370441 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:59.370408 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-ceac7b9278b8\"" Nov 02 23:09:59.370441 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:59.370437 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-8d45a4fafae3\"" Nov 02 23:09:59.406606 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:59.406569 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-88d4f7ff9-xsrng"] Nov 02 23:09:59.499062 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:59.499022 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eac149ce-3ee3-4232-8b82-10eb431d939d-routing-overlay-volume\") pod \"payload-processing-external-model-88d4f7ff9-xsrng\" (UID: \"eac149ce-3ee3-4232-8b82-10eb431d939d\") " pod="models-as-a-service/payload-processing-external-model-88d4f7ff9-xsrng" Nov 02 23:09:59.499326 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:59.499300 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eac149ce-3ee3-4232-8b82-10eb431d939d-provider-credentials\") pod \"payload-processing-external-model-88d4f7ff9-xsrng\" (UID: \"eac149ce-3ee3-4232-8b82-10eb431d939d\") " pod="models-as-a-service/payload-processing-external-model-88d4f7ff9-xsrng" Nov 02 23:09:59.504911 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:59.499539 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjzlr\" (UniqueName: \"kubernetes.io/projected/eac149ce-3ee3-4232-8b82-10eb431d939d-kube-api-access-kjzlr\") pod \"payload-processing-external-model-88d4f7ff9-xsrng\" (UID: \"eac149ce-3ee3-4232-8b82-10eb431d939d\") " pod="models-as-a-service/payload-processing-external-model-88d4f7ff9-xsrng" Nov 02 23:09:59.505230 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:59.505209 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eac149ce-3ee3-4232-8b82-10eb431d939d-plugins-config-volume\") pod \"payload-processing-external-model-88d4f7ff9-xsrng\" (UID: \"eac149ce-3ee3-4232-8b82-10eb431d939d\") " pod="models-as-a-service/payload-processing-external-model-88d4f7ff9-xsrng" Nov 02 23:09:59.606345 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:59.606257 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eac149ce-3ee3-4232-8b82-10eb431d939d-routing-overlay-volume\") pod \"payload-processing-external-model-88d4f7ff9-xsrng\" (UID: \"eac149ce-3ee3-4232-8b82-10eb431d939d\") " pod="models-as-a-service/payload-processing-external-model-88d4f7ff9-xsrng" Nov 02 23:09:59.606345 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:59.606295 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eac149ce-3ee3-4232-8b82-10eb431d939d-provider-credentials\") pod \"payload-processing-external-model-88d4f7ff9-xsrng\" (UID: \"eac149ce-3ee3-4232-8b82-10eb431d939d\") " pod="models-as-a-service/payload-processing-external-model-88d4f7ff9-xsrng" Nov 02 23:09:59.606345 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:59.606340 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjzlr\" (UniqueName: \"kubernetes.io/projected/eac149ce-3ee3-4232-8b82-10eb431d939d-kube-api-access-kjzlr\") pod \"payload-processing-external-model-88d4f7ff9-xsrng\" (UID: \"eac149ce-3ee3-4232-8b82-10eb431d939d\") " pod="models-as-a-service/payload-processing-external-model-88d4f7ff9-xsrng" Nov 02 23:09:59.606559 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:59.606396 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eac149ce-3ee3-4232-8b82-10eb431d939d-plugins-config-volume\") pod \"payload-processing-external-model-88d4f7ff9-xsrng\" (UID: \"eac149ce-3ee3-4232-8b82-10eb431d939d\") " pod="models-as-a-service/payload-processing-external-model-88d4f7ff9-xsrng" Nov 02 23:09:59.607075 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:59.607047 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eac149ce-3ee3-4232-8b82-10eb431d939d-routing-overlay-volume\") pod \"payload-processing-external-model-88d4f7ff9-xsrng\" (UID: \"eac149ce-3ee3-4232-8b82-10eb431d939d\") " pod="models-as-a-service/payload-processing-external-model-88d4f7ff9-xsrng" Nov 02 23:09:59.607207 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:59.607150 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eac149ce-3ee3-4232-8b82-10eb431d939d-plugins-config-volume\") pod \"payload-processing-external-model-88d4f7ff9-xsrng\" (UID: \"eac149ce-3ee3-4232-8b82-10eb431d939d\") " pod="models-as-a-service/payload-processing-external-model-88d4f7ff9-xsrng" Nov 02 23:09:59.608854 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:59.608835 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eac149ce-3ee3-4232-8b82-10eb431d939d-provider-credentials\") pod \"payload-processing-external-model-88d4f7ff9-xsrng\" (UID: \"eac149ce-3ee3-4232-8b82-10eb431d939d\") " pod="models-as-a-service/payload-processing-external-model-88d4f7ff9-xsrng" Nov 02 23:09:59.660970 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:59.660945 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjzlr\" (UniqueName: \"kubernetes.io/projected/eac149ce-3ee3-4232-8b82-10eb431d939d-kube-api-access-kjzlr\") pod \"payload-processing-external-model-88d4f7ff9-xsrng\" (UID: \"eac149ce-3ee3-4232-8b82-10eb431d939d\") " pod="models-as-a-service/payload-processing-external-model-88d4f7ff9-xsrng" Nov 02 23:09:59.933243 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:09:59.933165 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-88d4f7ff9-xsrng" Nov 02 23:10:00.304605 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:00.304572 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-88d4f7ff9-xsrng"] Nov 02 23:10:00.305034 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:10:00.305006 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeac149ce_3ee3_4232_8b82_10eb431d939d.slice/crio-f53195dbb2988b2d78768cdc86649b963576dedb15f4121464d76927d1f013a0 WatchSource:0}: Error finding container f53195dbb2988b2d78768cdc86649b963576dedb15f4121464d76927d1f013a0: Status 404 returned error can't find the container with id f53195dbb2988b2d78768cdc86649b963576dedb15f4121464d76927d1f013a0 Nov 02 23:10:00.402572 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:00.402539 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg"] Nov 02 23:10:00.406365 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:00.406344 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" Nov 02 23:10:00.416466 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:00.416443 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8027a0-9041-42ec-af51-46526e00811f-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg\" (UID: \"dc8027a0-9041-42ec-af51-46526e00811f\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" Nov 02 23:10:00.416561 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:00.416477 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc8027a0-9041-42ec-af51-46526e00811f-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg\" (UID: \"dc8027a0-9041-42ec-af51-46526e00811f\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" Nov 02 23:10:00.416561 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:00.416495 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4krqb\" (UniqueName: \"kubernetes.io/projected/dc8027a0-9041-42ec-af51-46526e00811f-kube-api-access-4krqb\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg\" (UID: \"dc8027a0-9041-42ec-af51-46526e00811f\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" Nov 02 23:10:00.416646 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:00.416606 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc8027a0-9041-42ec-af51-46526e00811f-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg\" (UID: \"dc8027a0-9041-42ec-af51-46526e00811f\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" Nov 02 23:10:00.416646 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:00.416629 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc8027a0-9041-42ec-af51-46526e00811f-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg\" (UID: \"dc8027a0-9041-42ec-af51-46526e00811f\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" Nov 02 23:10:00.416715 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:00.416672 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8027a0-9041-42ec-af51-46526e00811f-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg\" (UID: \"dc8027a0-9041-42ec-af51-46526e00811f\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" Nov 02 23:10:00.416748 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:00.416715 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc8027a0-9041-42ec-af51-46526e00811f-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg\" (UID: \"dc8027a0-9041-42ec-af51-46526e00811f\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" Nov 02 23:10:00.416789 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:00.416770 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc8027a0-9041-42ec-af51-46526e00811f-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg\" (UID: \"dc8027a0-9041-42ec-af51-46526e00811f\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" Nov 02 23:10:00.416826 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:00.416797 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc8027a0-9041-42ec-af51-46526e00811f-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg\" (UID: \"dc8027a0-9041-42ec-af51-46526e00811f\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" Nov 02 23:10:00.464217 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:00.464182 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg"] Nov 02 23:10:00.519107 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:00.518948 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc8027a0-9041-42ec-af51-46526e00811f-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg\" (UID: \"dc8027a0-9041-42ec-af51-46526e00811f\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" Nov 02 23:10:00.519107 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:00.519015 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc8027a0-9041-42ec-af51-46526e00811f-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg\" (UID: \"dc8027a0-9041-42ec-af51-46526e00811f\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" Nov 02 23:10:00.519447 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:00.519415 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc8027a0-9041-42ec-af51-46526e00811f-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg\" (UID: \"dc8027a0-9041-42ec-af51-46526e00811f\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" Nov 02 23:10:00.519616 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:00.519414 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8027a0-9041-42ec-af51-46526e00811f-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg\" (UID: \"dc8027a0-9041-42ec-af51-46526e00811f\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" Nov 02 23:10:00.519616 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:00.519500 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc8027a0-9041-42ec-af51-46526e00811f-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg\" (UID: \"dc8027a0-9041-42ec-af51-46526e00811f\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" Nov 02 23:10:00.519616 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:00.519527 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4krqb\" (UniqueName: \"kubernetes.io/projected/dc8027a0-9041-42ec-af51-46526e00811f-kube-api-access-4krqb\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg\" (UID: \"dc8027a0-9041-42ec-af51-46526e00811f\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" Nov 02 23:10:00.519616 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:00.519584 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc8027a0-9041-42ec-af51-46526e00811f-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg\" (UID: \"dc8027a0-9041-42ec-af51-46526e00811f\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" Nov 02 23:10:00.519867 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:00.519617 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc8027a0-9041-42ec-af51-46526e00811f-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg\" (UID: \"dc8027a0-9041-42ec-af51-46526e00811f\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" Nov 02 23:10:00.519867 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:00.519696 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8027a0-9041-42ec-af51-46526e00811f-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg\" (UID: \"dc8027a0-9041-42ec-af51-46526e00811f\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" Nov 02 23:10:00.519867 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:00.519724 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc8027a0-9041-42ec-af51-46526e00811f-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg\" (UID: \"dc8027a0-9041-42ec-af51-46526e00811f\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" Nov 02 23:10:00.519867 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:00.519765 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8027a0-9041-42ec-af51-46526e00811f-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg\" (UID: \"dc8027a0-9041-42ec-af51-46526e00811f\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" Nov 02 23:10:00.520105 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:00.520070 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8027a0-9041-42ec-af51-46526e00811f-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg\" (UID: \"dc8027a0-9041-42ec-af51-46526e00811f\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" Nov 02 23:10:00.520170 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:00.520125 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc8027a0-9041-42ec-af51-46526e00811f-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg\" (UID: \"dc8027a0-9041-42ec-af51-46526e00811f\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" Nov 02 23:10:00.520760 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:00.520735 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc8027a0-9041-42ec-af51-46526e00811f-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg\" (UID: \"dc8027a0-9041-42ec-af51-46526e00811f\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" Nov 02 23:10:00.522557 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:00.522537 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc8027a0-9041-42ec-af51-46526e00811f-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg\" (UID: \"dc8027a0-9041-42ec-af51-46526e00811f\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" Nov 02 23:10:00.522938 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:00.522917 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc8027a0-9041-42ec-af51-46526e00811f-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg\" (UID: \"dc8027a0-9041-42ec-af51-46526e00811f\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" Nov 02 23:10:00.574119 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:00.574049 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc8027a0-9041-42ec-af51-46526e00811f-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg\" (UID: \"dc8027a0-9041-42ec-af51-46526e00811f\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" Nov 02 23:10:00.574253 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:00.574156 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4krqb\" (UniqueName: \"kubernetes.io/projected/dc8027a0-9041-42ec-af51-46526e00811f-kube-api-access-4krqb\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg\" (UID: \"dc8027a0-9041-42ec-af51-46526e00811f\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" Nov 02 23:10:00.719631 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:00.719590 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" Nov 02 23:10:00.819593 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:00.819558 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-88d4f7ff9-xsrng" event={"ID":"eac149ce-3ee3-4232-8b82-10eb431d939d","Type":"ContainerStarted","Data":"7151f5dc86ba539512a5b553a4d76ec2d3db32a403eed92beb678d25f1404af0"} Nov 02 23:10:00.819776 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:00.819622 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-88d4f7ff9-xsrng" event={"ID":"eac149ce-3ee3-4232-8b82-10eb431d939d","Type":"ContainerStarted","Data":"f53195dbb2988b2d78768cdc86649b963576dedb15f4121464d76927d1f013a0"} Nov 02 23:10:00.819776 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:00.819661 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-88d4f7ff9-xsrng" Nov 02 23:10:00.930016 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:00.929884 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-88d4f7ff9-xsrng" podStartSLOduration=1.929869594 podStartE2EDuration="1.929869594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:10:00.929017604 +0000 UTC m=+1049.897549392" watchObservedRunningTime="2026-11-02 23:10:00.929869594 +0000 UTC m=+1049.898401379" Nov 02 23:10:01.137525 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:01.137496 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg"] Nov 02 23:10:01.138984 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:10:01.138955 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc8027a0_9041_42ec_af51_46526e00811f.slice/crio-d9ea71ace16e2630edeeeafd9fffdded8e7fdbe3eed15d6c592600c52650d5fb WatchSource:0}: Error finding container d9ea71ace16e2630edeeeafd9fffdded8e7fdbe3eed15d6c592600c52650d5fb: Status 404 returned error can't find the container with id d9ea71ace16e2630edeeeafd9fffdded8e7fdbe3eed15d6c592600c52650d5fb Nov 02 23:10:01.141560 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:01.141526 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:10:01.141675 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:01.141593 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:10:01.141675 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:01.141645 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:10:01.843551 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:01.843499 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" event={"ID":"dc8027a0-9041-42ec-af51-46526e00811f","Type":"ContainerStarted","Data":"51f36e5dc64ab48ba1c7ad1815bd0d6fa057350293cdabaa761e8a07c9cefed7"} Nov 02 23:10:01.844069 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:01.843573 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" event={"ID":"dc8027a0-9041-42ec-af51-46526e00811f","Type":"ContainerStarted","Data":"d9ea71ace16e2630edeeeafd9fffdded8e7fdbe3eed15d6c592600c52650d5fb"} Nov 02 23:10:02.052040 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:02.051975 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" podStartSLOduration=2.051961849 podStartE2EDuration="2.051961849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:10:02.00370519 +0000 UTC m=+1050.972236974" watchObservedRunningTime="2026-11-02 23:10:02.051961849 +0000 UTC m=+1051.020493634" Nov 02 23:10:02.720616 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:02.720579 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" Nov 02 23:10:02.733910 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:02.731019 5076 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 02 23:10:02.733910 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:02.731082 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" podUID="dc8027a0-9041-42ec-af51-46526e00811f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 02 23:10:03.724915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:03.724289 5076 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 02 23:10:03.724915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:03.724352 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" podUID="dc8027a0-9041-42ec-af51-46526e00811f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 02 23:10:04.720915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:04.720334 5076 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 02 23:10:04.720915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:04.720395 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" podUID="dc8027a0-9041-42ec-af51-46526e00811f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 02 23:10:04.952915 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:10:04.952835 5076 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ef36b3c11cbb: secret "agc-cred-ef36b3c11cbb" not found Nov 02 23:10:04.952915 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:10:04.952869 5076 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5f5b4df8c-8kdbn: secret "agc-cred-ef36b3c11cbb" not found Nov 02 23:10:04.986976 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:10:04.969031 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8b59cd8-c08c-43af-9345-edba8a21e667-provider-credentials podName:f8b59cd8-c08c-43af-9345-edba8a21e667 nodeName:}" failed. No retries permitted until 2026-11-02 23:10:20.968982246 +0000 UTC m=+1069.937514024 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f8b59cd8-c08c-43af-9345-edba8a21e667-provider-credentials") pod "payload-processing-external-model-5f5b4df8c-8kdbn" (UID: "f8b59cd8-c08c-43af-9345-edba8a21e667") : secret "agc-cred-ef36b3c11cbb" not found Nov 02 23:10:05.730915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:05.728948 5076 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 02 23:10:05.730915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:05.729007 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" podUID="dc8027a0-9041-42ec-af51-46526e00811f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 02 23:10:06.722915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:06.722435 5076 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 02 23:10:06.722915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:06.722519 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" podUID="dc8027a0-9041-42ec-af51-46526e00811f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 02 23:10:07.721929 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:07.721305 5076 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 02 23:10:07.721929 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:07.721368 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" podUID="dc8027a0-9041-42ec-af51-46526e00811f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 02 23:10:09.237350 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:09.237314 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" Nov 02 23:10:09.237841 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:09.237693 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" Nov 02 23:10:09.238491 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:09.238473 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" Nov 02 23:10:09.883593 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:10:09.883560 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:10:11.847305 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:11.847271 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-88d4f7ff9-xsrng" Nov 02 23:10:12.011346 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:12.011313 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f5b4df8c-8kdbn"] Nov 02 23:10:17.012064 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:17.011997 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5f5b4df8c-8kdbn" podUID="f8b59cd8-c08c-43af-9345-edba8a21e667" containerName="payload-processing" containerID="cri-o://a41f06ae95e1c27ba603cd825519f500cb58132c86524d7e261356d05dafddd0" gracePeriod=25 Nov 02 23:10:17.267475 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:17.267410 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5f5b4df8c-8kdbn" Nov 02 23:10:17.275674 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:17.275647 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8b59cd8-c08c-43af-9345-edba8a21e667-kube-api-access-gtr4c\" (UniqueName: \"kubernetes.io/projected/f8b59cd8-c08c-43af-9345-edba8a21e667-kube-api-access-gtr4c\") pod \"f8b59cd8-c08c-43af-9345-edba8a21e667\" (UID: \"f8b59cd8-c08c-43af-9345-edba8a21e667\") " Nov 02 23:10:17.275826 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:17.275767 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8b59cd8-c08c-43af-9345-edba8a21e667-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f8b59cd8-c08c-43af-9345-edba8a21e667-provider-credentials\") pod \"f8b59cd8-c08c-43af-9345-edba8a21e667\" (UID: \"f8b59cd8-c08c-43af-9345-edba8a21e667\") " Nov 02 23:10:17.275935 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:17.275921 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8b59cd8-c08c-43af-9345-edba8a21e667-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f8b59cd8-c08c-43af-9345-edba8a21e667-routing-overlay-volume\") pod \"f8b59cd8-c08c-43af-9345-edba8a21e667\" (UID: \"f8b59cd8-c08c-43af-9345-edba8a21e667\") " Nov 02 23:10:17.275995 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:17.275983 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8b59cd8-c08c-43af-9345-edba8a21e667-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8b59cd8-c08c-43af-9345-edba8a21e667-plugins-config-volume\") pod \"f8b59cd8-c08c-43af-9345-edba8a21e667\" (UID: \"f8b59cd8-c08c-43af-9345-edba8a21e667\") " Nov 02 23:10:17.276282 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:17.276255 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8b59cd8-c08c-43af-9345-edba8a21e667-routing-overlay-volume" pod "f8b59cd8-c08c-43af-9345-edba8a21e667" (UID: "f8b59cd8-c08c-43af-9345-edba8a21e667"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:10:17.276414 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:17.276373 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8b59cd8-c08c-43af-9345-edba8a21e667-plugins-config-volume" pod "f8b59cd8-c08c-43af-9345-edba8a21e667" (UID: "f8b59cd8-c08c-43af-9345-edba8a21e667"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:10:17.276486 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:17.276455 5076 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f8b59cd8-c08c-43af-9345-edba8a21e667-routing-overlay-volume\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:10:17.276486 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:17.276469 5076 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8b59cd8-c08c-43af-9345-edba8a21e667-plugins-config-volume\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:10:17.277877 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:17.277854 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8b59cd8-c08c-43af-9345-edba8a21e667-provider-credentials" pod "f8b59cd8-c08c-43af-9345-edba8a21e667" (UID: "f8b59cd8-c08c-43af-9345-edba8a21e667"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:10:17.278032 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:17.277975 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8b59cd8-c08c-43af-9345-edba8a21e667-kube-api-access-gtr4c" pod "f8b59cd8-c08c-43af-9345-edba8a21e667" (UID: "f8b59cd8-c08c-43af-9345-edba8a21e667"). InnerVolumeSpecName "kube-api-access-gtr4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:10:17.377677 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:17.377639 5076 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f8b59cd8-c08c-43af-9345-edba8a21e667-provider-credentials\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:10:17.377677 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:17.377678 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtr4c\" (UniqueName: \"kubernetes.io/projected/f8b59cd8-c08c-43af-9345-edba8a21e667-kube-api-access-gtr4c\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:10:18.064594 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:18.064561 5076 generic.go:356] "Generic (PLEG): container finished" podID="f8b59cd8-c08c-43af-9345-edba8a21e667" containerID="a41f06ae95e1c27ba603cd825519f500cb58132c86524d7e261356d05dafddd0" exitCode=0 Nov 02 23:10:18.065147 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:18.064601 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5f5b4df8c-8kdbn" event={"ID":"f8b59cd8-c08c-43af-9345-edba8a21e667","Type":"ContainerDied","Data":"a41f06ae95e1c27ba603cd825519f500cb58132c86524d7e261356d05dafddd0"} Nov 02 23:10:18.065147 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:18.064608 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5f5b4df8c-8kdbn" Nov 02 23:10:18.065147 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:18.064640 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5f5b4df8c-8kdbn" event={"ID":"f8b59cd8-c08c-43af-9345-edba8a21e667","Type":"ContainerDied","Data":"0517efaa9f5b0f1781198ea32c080b8df21846533fe260d12f4d386f954f391d"} Nov 02 23:10:18.065147 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:18.064662 5076 scope.go:122] "RemoveContainer" containerID="a41f06ae95e1c27ba603cd825519f500cb58132c86524d7e261356d05dafddd0" Nov 02 23:10:18.074570 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:18.074552 5076 scope.go:122] "RemoveContainer" containerID="a41f06ae95e1c27ba603cd825519f500cb58132c86524d7e261356d05dafddd0" Nov 02 23:10:18.074839 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:10:18.074821 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a41f06ae95e1c27ba603cd825519f500cb58132c86524d7e261356d05dafddd0\": container with ID starting with a41f06ae95e1c27ba603cd825519f500cb58132c86524d7e261356d05dafddd0 not found: ID does not exist" containerID="a41f06ae95e1c27ba603cd825519f500cb58132c86524d7e261356d05dafddd0" Nov 02 23:10:18.074881 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:18.074848 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a41f06ae95e1c27ba603cd825519f500cb58132c86524d7e261356d05dafddd0"} err="failed to get container status \"a41f06ae95e1c27ba603cd825519f500cb58132c86524d7e261356d05dafddd0\": rpc error: code = NotFound desc = could not find container \"a41f06ae95e1c27ba603cd825519f500cb58132c86524d7e261356d05dafddd0\": container with ID starting with a41f06ae95e1c27ba603cd825519f500cb58132c86524d7e261356d05dafddd0 not found: ID does not exist" Nov 02 23:10:18.222213 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:18.222179 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f5b4df8c-8kdbn"] Nov 02 23:10:18.226282 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:18.226245 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f5b4df8c-8kdbn"] Nov 02 23:10:19.148777 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:19.148746 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8b59cd8-c08c-43af-9345-edba8a21e667" path="/var/lib/kubelet/pods/f8b59cd8-c08c-43af-9345-edba8a21e667/volumes" Nov 02 23:10:31.346473 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:10:31.346438 5076 scope.go:122] "RemoveContainer" containerID="2f7c0ca7d96e9c02d1ec1fc285ac921b8f62e66456638404450a58729df3185e" Nov 02 23:10:39.884314 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:10:39.884279 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:11:09.885251 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:11:09.885201 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:11:26.974715 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:26.974678 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7"] Nov 02 23:11:26.975288 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:26.975272 5076 cpu_manager.go:414] "containerMap: removing container" podUID="f8b59cd8-c08c-43af-9345-edba8a21e667" containerName="payload-processing" Nov 02 23:11:26.975333 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:26.975289 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8b59cd8-c08c-43af-9345-edba8a21e667" containerName="payload-processing" Nov 02 23:11:26.975408 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:26.975399 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8b59cd8-c08c-43af-9345-edba8a21e667" containerName="payload-processing" Nov 02 23:11:26.978998 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:26.978973 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" Nov 02 23:11:27.008499 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:27.008466 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3751da0e-99b6-4f2c-836e-699267f1428d-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7\" (UID: \"3751da0e-99b6-4f2c-836e-699267f1428d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" Nov 02 23:11:27.008665 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:27.008523 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3751da0e-99b6-4f2c-836e-699267f1428d-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7\" (UID: \"3751da0e-99b6-4f2c-836e-699267f1428d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" Nov 02 23:11:27.008741 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:27.008655 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3751da0e-99b6-4f2c-836e-699267f1428d-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7\" (UID: \"3751da0e-99b6-4f2c-836e-699267f1428d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" Nov 02 23:11:27.008741 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:27.008715 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvvsl\" (UniqueName: \"kubernetes.io/projected/3751da0e-99b6-4f2c-836e-699267f1428d-kube-api-access-cvvsl\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7\" (UID: \"3751da0e-99b6-4f2c-836e-699267f1428d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" Nov 02 23:11:27.008840 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:27.008822 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3751da0e-99b6-4f2c-836e-699267f1428d-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7\" (UID: \"3751da0e-99b6-4f2c-836e-699267f1428d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" Nov 02 23:11:27.008875 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:27.008851 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3751da0e-99b6-4f2c-836e-699267f1428d-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7\" (UID: \"3751da0e-99b6-4f2c-836e-699267f1428d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" Nov 02 23:11:27.008961 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:27.008948 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3751da0e-99b6-4f2c-836e-699267f1428d-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7\" (UID: \"3751da0e-99b6-4f2c-836e-699267f1428d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" Nov 02 23:11:27.009007 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:27.008972 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3751da0e-99b6-4f2c-836e-699267f1428d-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7\" (UID: \"3751da0e-99b6-4f2c-836e-699267f1428d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" Nov 02 23:11:27.009056 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:27.009007 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3751da0e-99b6-4f2c-836e-699267f1428d-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7\" (UID: \"3751da0e-99b6-4f2c-836e-699267f1428d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" Nov 02 23:11:27.082137 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:27.082102 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7"] Nov 02 23:11:27.109647 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:27.109612 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3751da0e-99b6-4f2c-836e-699267f1428d-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7\" (UID: \"3751da0e-99b6-4f2c-836e-699267f1428d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" Nov 02 23:11:27.109647 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:27.109647 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3751da0e-99b6-4f2c-836e-699267f1428d-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7\" (UID: \"3751da0e-99b6-4f2c-836e-699267f1428d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" Nov 02 23:11:27.109929 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:27.109682 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3751da0e-99b6-4f2c-836e-699267f1428d-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7\" (UID: \"3751da0e-99b6-4f2c-836e-699267f1428d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" Nov 02 23:11:27.109929 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:27.109709 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3751da0e-99b6-4f2c-836e-699267f1428d-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7\" (UID: \"3751da0e-99b6-4f2c-836e-699267f1428d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" Nov 02 23:11:27.109929 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:27.109748 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3751da0e-99b6-4f2c-836e-699267f1428d-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7\" (UID: \"3751da0e-99b6-4f2c-836e-699267f1428d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" Nov 02 23:11:27.109929 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:27.109812 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3751da0e-99b6-4f2c-836e-699267f1428d-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7\" (UID: \"3751da0e-99b6-4f2c-836e-699267f1428d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" Nov 02 23:11:27.109929 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:27.109854 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvvsl\" (UniqueName: \"kubernetes.io/projected/3751da0e-99b6-4f2c-836e-699267f1428d-kube-api-access-cvvsl\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7\" (UID: \"3751da0e-99b6-4f2c-836e-699267f1428d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" Nov 02 23:11:27.110208 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:27.109957 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3751da0e-99b6-4f2c-836e-699267f1428d-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7\" (UID: \"3751da0e-99b6-4f2c-836e-699267f1428d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" Nov 02 23:11:27.110208 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:27.109986 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3751da0e-99b6-4f2c-836e-699267f1428d-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7\" (UID: \"3751da0e-99b6-4f2c-836e-699267f1428d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" Nov 02 23:11:27.110324 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:27.110249 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3751da0e-99b6-4f2c-836e-699267f1428d-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7\" (UID: \"3751da0e-99b6-4f2c-836e-699267f1428d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" Nov 02 23:11:27.110379 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:27.110313 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3751da0e-99b6-4f2c-836e-699267f1428d-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7\" (UID: \"3751da0e-99b6-4f2c-836e-699267f1428d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" Nov 02 23:11:27.110433 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:27.110396 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3751da0e-99b6-4f2c-836e-699267f1428d-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7\" (UID: \"3751da0e-99b6-4f2c-836e-699267f1428d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" Nov 02 23:11:27.110433 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:27.110400 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3751da0e-99b6-4f2c-836e-699267f1428d-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7\" (UID: \"3751da0e-99b6-4f2c-836e-699267f1428d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" Nov 02 23:11:27.110433 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:27.110424 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3751da0e-99b6-4f2c-836e-699267f1428d-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7\" (UID: \"3751da0e-99b6-4f2c-836e-699267f1428d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" Nov 02 23:11:27.112191 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:27.112163 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3751da0e-99b6-4f2c-836e-699267f1428d-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7\" (UID: \"3751da0e-99b6-4f2c-836e-699267f1428d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" Nov 02 23:11:27.112477 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:27.112459 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3751da0e-99b6-4f2c-836e-699267f1428d-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7\" (UID: \"3751da0e-99b6-4f2c-836e-699267f1428d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" Nov 02 23:11:27.165514 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:27.165481 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3751da0e-99b6-4f2c-836e-699267f1428d-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7\" (UID: \"3751da0e-99b6-4f2c-836e-699267f1428d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" Nov 02 23:11:27.165670 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:27.165629 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvvsl\" (UniqueName: \"kubernetes.io/projected/3751da0e-99b6-4f2c-836e-699267f1428d-kube-api-access-cvvsl\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7\" (UID: \"3751da0e-99b6-4f2c-836e-699267f1428d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" Nov 02 23:11:27.292314 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:27.292233 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" Nov 02 23:11:27.710258 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:27.710229 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7"] Nov 02 23:11:27.710945 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:11:27.710915 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3751da0e_99b6_4f2c_836e_699267f1428d.slice/crio-4c3b0fc779ee2de7c5544a0b38b6ed949ee52c31b86872cfa62f7d7630d2a951 WatchSource:0}: Error finding container 4c3b0fc779ee2de7c5544a0b38b6ed949ee52c31b86872cfa62f7d7630d2a951: Status 404 returned error can't find the container with id 4c3b0fc779ee2de7c5544a0b38b6ed949ee52c31b86872cfa62f7d7630d2a951 Nov 02 23:11:27.713201 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:27.713169 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:11:27.713280 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:27.713231 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:11:27.713280 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:27.713258 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:11:28.272615 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:28.272577 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f"] Nov 02 23:11:28.276555 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:28.276530 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" Nov 02 23:11:28.340657 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:28.340615 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-credential-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f\" (UID: \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" Nov 02 23:11:28.340836 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:28.340677 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-istio-podinfo\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f\" (UID: \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" Nov 02 23:11:28.340836 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:28.340710 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2pvj\" (UniqueName: \"kubernetes.io/projected/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-kube-api-access-j2pvj\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f\" (UID: \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" Nov 02 23:11:28.340836 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:28.340739 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-istio-data\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f\" (UID: \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" Nov 02 23:11:28.341077 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:28.340851 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-istio-token\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f\" (UID: \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" Nov 02 23:11:28.341077 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:28.340881 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-workload-certs\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f\" (UID: \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" Nov 02 23:11:28.341185 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:28.341086 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-istio-envoy\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f\" (UID: \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" Nov 02 23:11:28.341185 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:28.341122 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-istiod-ca-cert\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f\" (UID: \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" Nov 02 23:11:28.341293 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:28.341185 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-workload-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f\" (UID: \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" Nov 02 23:11:28.389868 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:28.389826 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f"] Nov 02 23:11:28.442804 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:28.442764 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-credential-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f\" (UID: \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" Nov 02 23:11:28.443014 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:28.442815 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-istio-podinfo\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f\" (UID: \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" Nov 02 23:11:28.443014 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:28.442845 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2pvj\" (UniqueName: \"kubernetes.io/projected/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-kube-api-access-j2pvj\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f\" (UID: \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" Nov 02 23:11:28.443014 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:28.442872 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-istio-data\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f\" (UID: \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" Nov 02 23:11:28.443014 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:28.442988 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-istio-token\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f\" (UID: \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" Nov 02 23:11:28.443222 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:28.443017 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-workload-certs\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f\" (UID: \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" Nov 02 23:11:28.443222 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:28.443072 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-istio-envoy\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f\" (UID: \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" Nov 02 23:11:28.443222 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:28.443097 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-istiod-ca-cert\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f\" (UID: \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" Nov 02 23:11:28.443222 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:28.443158 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-workload-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f\" (UID: \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" Nov 02 23:11:28.443422 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:28.443278 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-credential-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f\" (UID: \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" Nov 02 23:11:28.443422 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:28.443295 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-istio-data\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f\" (UID: \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" Nov 02 23:11:28.443677 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:28.443656 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-workload-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f\" (UID: \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" Nov 02 23:11:28.443770 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:28.443745 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-workload-certs\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f\" (UID: \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" Nov 02 23:11:28.443987 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:28.443968 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-istiod-ca-cert\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f\" (UID: \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" Nov 02 23:11:28.445776 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:28.445753 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-istio-envoy\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f\" (UID: \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" Nov 02 23:11:28.446037 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:28.446021 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-istio-podinfo\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f\" (UID: \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" Nov 02 23:11:28.499420 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:28.499389 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-istio-token\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f\" (UID: \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" Nov 02 23:11:28.499672 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:28.499650 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2pvj\" (UniqueName: \"kubernetes.io/projected/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-kube-api-access-j2pvj\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f\" (UID: \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" Nov 02 23:11:28.591090 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:28.590988 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" Nov 02 23:11:28.649497 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:28.649456 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" event={"ID":"3751da0e-99b6-4f2c-836e-699267f1428d","Type":"ContainerStarted","Data":"9f9e7a539b3f290115ff21f86c5bfe2ade555ff64c7699eee91e34c4b032447c"} Nov 02 23:11:28.649685 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:28.649516 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" event={"ID":"3751da0e-99b6-4f2c-836e-699267f1428d","Type":"ContainerStarted","Data":"4c3b0fc779ee2de7c5544a0b38b6ed949ee52c31b86872cfa62f7d7630d2a951"} Nov 02 23:11:28.763124 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:28.763061 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" podStartSLOduration=2.763042773 podStartE2EDuration="2.763042773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:11:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:11:28.761051981 +0000 UTC m=+1137.729583766" watchObservedRunningTime="2026-11-02 23:11:28.763042773 +0000 UTC m=+1137.731574559" Nov 02 23:11:29.038159 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:29.038116 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f"] Nov 02 23:11:29.041810 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:11:29.041774 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccc8ac87_3bdb_4fad_8bb5_5ccefd0a357b.slice/crio-eb55c978a06dd62e8fdfdd61a1a86657ea2cd382ff7ca5a1eedd268626e04f9d WatchSource:0}: Error finding container eb55c978a06dd62e8fdfdd61a1a86657ea2cd382ff7ca5a1eedd268626e04f9d: Status 404 returned error can't find the container with id eb55c978a06dd62e8fdfdd61a1a86657ea2cd382ff7ca5a1eedd268626e04f9d Nov 02 23:11:29.044318 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:29.044289 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:11:29.044412 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:29.044358 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:11:29.044412 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:29.044391 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:11:29.293255 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:29.293170 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" Nov 02 23:11:29.294777 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:29.294747 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 02 23:11:29.295197 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:29.294796 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" podUID="3751da0e-99b6-4f2c-836e-699267f1428d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 02 23:11:29.665193 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:29.664946 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" event={"ID":"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b","Type":"ContainerStarted","Data":"49abda58f880b7c522b788b8ee508aee1136589b658e424dea48ca2cec37fdad"} Nov 02 23:11:29.665193 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:29.665002 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" event={"ID":"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b","Type":"ContainerStarted","Data":"eb55c978a06dd62e8fdfdd61a1a86657ea2cd382ff7ca5a1eedd268626e04f9d"} Nov 02 23:11:29.790322 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:29.790271 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" podStartSLOduration=1.790256098 podStartE2EDuration="1.790256098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:11:29.787368103 +0000 UTC m=+1138.755899892" watchObservedRunningTime="2026-11-02 23:11:29.790256098 +0000 UTC m=+1138.758787951" Nov 02 23:11:30.293485 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:30.293448 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 02 23:11:30.293919 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:30.293528 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" podUID="3751da0e-99b6-4f2c-836e-699267f1428d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 02 23:11:30.591694 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:30.591599 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" Nov 02 23:11:30.593124 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:30.593100 5076 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 02 23:11:30.593206 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:30.593153 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" podUID="ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 02 23:11:31.292699 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:31.292669 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 02 23:11:31.292918 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:31.292714 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" podUID="3751da0e-99b6-4f2c-836e-699267f1428d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 02 23:11:31.591444 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:31.591362 5076 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 02 23:11:31.591444 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:31.591410 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" podUID="ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 02 23:11:32.293022 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:32.292989 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 02 23:11:32.293192 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:32.293040 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" podUID="3751da0e-99b6-4f2c-836e-699267f1428d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 02 23:11:32.591851 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:32.591771 5076 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 02 23:11:32.592395 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:32.591979 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" podUID="ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 02 23:11:33.292998 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:33.292967 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 02 23:11:33.293180 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:33.293017 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" podUID="3751da0e-99b6-4f2c-836e-699267f1428d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 02 23:11:33.591448 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:33.591366 5076 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 02 23:11:33.591448 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:33.591413 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" podUID="ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 02 23:11:34.293154 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:34.293117 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 02 23:11:34.293558 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:34.293176 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" podUID="3751da0e-99b6-4f2c-836e-699267f1428d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 02 23:11:34.592091 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:34.591994 5076 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 02 23:11:34.592091 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:34.592049 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" podUID="ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 02 23:11:35.297042 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:35.297008 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" Nov 02 23:11:35.297413 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:35.297377 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" Nov 02 23:11:35.297993 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:35.297971 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" Nov 02 23:11:35.592337 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:35.592244 5076 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 02 23:11:35.592337 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:35.592309 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" podUID="ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 02 23:11:36.596042 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:36.596009 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" Nov 02 23:11:36.596519 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:36.596400 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" Nov 02 23:11:36.597080 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:36.597057 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" Nov 02 23:11:39.886807 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:11:39.886770 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:11:57.215184 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:57.215151 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8"] Nov 02 23:11:57.218372 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:57.218353 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" Nov 02 23:11:57.326481 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:57.326448 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8"] Nov 02 23:11:57.360595 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:57.360565 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8\" (UID: \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" Nov 02 23:11:57.360788 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:57.360617 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8\" (UID: \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" Nov 02 23:11:57.360788 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:57.360638 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8\" (UID: \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" Nov 02 23:11:57.360912 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:57.360776 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmdpq\" (UniqueName: \"kubernetes.io/projected/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-kube-api-access-wmdpq\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8\" (UID: \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" Nov 02 23:11:57.360912 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:57.360826 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8\" (UID: \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" Nov 02 23:11:57.360982 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:57.360912 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8\" (UID: \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" Nov 02 23:11:57.361015 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:57.360996 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8\" (UID: \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" Nov 02 23:11:57.361051 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:57.361030 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8\" (UID: \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" Nov 02 23:11:57.361103 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:57.361091 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8\" (UID: \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" Nov 02 23:11:57.462084 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:57.462051 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmdpq\" (UniqueName: \"kubernetes.io/projected/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-kube-api-access-wmdpq\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8\" (UID: \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" Nov 02 23:11:57.462288 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:57.462093 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8\" (UID: \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" Nov 02 23:11:57.462288 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:57.462115 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8\" (UID: \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" Nov 02 23:11:57.462288 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:57.462169 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8\" (UID: \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" Nov 02 23:11:57.462288 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:57.462189 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8\" (UID: \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" Nov 02 23:11:57.462288 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:57.462264 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8\" (UID: \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" Nov 02 23:11:57.462571 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:57.462298 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8\" (UID: \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" Nov 02 23:11:57.462571 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:57.462350 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8\" (UID: \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" Nov 02 23:11:57.462571 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:57.462378 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8\" (UID: \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" Nov 02 23:11:57.462960 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:57.462913 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8\" (UID: \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" Nov 02 23:11:57.463088 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:57.462968 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8\" (UID: \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" Nov 02 23:11:57.463088 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:57.463032 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8\" (UID: \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" Nov 02 23:11:57.463088 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:57.463036 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8\" (UID: \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" Nov 02 23:11:57.463261 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:57.463185 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8\" (UID: \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" Nov 02 23:11:57.464832 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:57.464806 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8\" (UID: \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" Nov 02 23:11:57.465036 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:57.465018 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8\" (UID: \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" Nov 02 23:11:57.516422 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:57.516352 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8\" (UID: \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" Nov 02 23:11:57.516550 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:57.516436 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmdpq\" (UniqueName: \"kubernetes.io/projected/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-kube-api-access-wmdpq\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8\" (UID: \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" Nov 02 23:11:57.530361 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:57.530334 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" Nov 02 23:11:57.944256 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:57.944227 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8"] Nov 02 23:11:57.946549 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:11:57.946521 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73f3edf1_31bb_41a5_abb8_8a97ab98b1b4.slice/crio-5db6b1c474e7b3d7541ebfeae476ab86de0e593ba1f6a63ed4900e58f9fcc45d WatchSource:0}: Error finding container 5db6b1c474e7b3d7541ebfeae476ab86de0e593ba1f6a63ed4900e58f9fcc45d: Status 404 returned error can't find the container with id 5db6b1c474e7b3d7541ebfeae476ab86de0e593ba1f6a63ed4900e58f9fcc45d Nov 02 23:11:57.948755 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:57.948719 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:11:57.948826 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:57.948786 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:11:57.948826 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:57.948814 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:11:58.848921 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:58.848866 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" event={"ID":"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4","Type":"ContainerStarted","Data":"dba449aa9a5e3ef461583ed7bdd9e17a1ee3f32938ae213fb4a53998b535a5fa"} Nov 02 23:11:58.849329 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:58.848939 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" event={"ID":"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4","Type":"ContainerStarted","Data":"5db6b1c474e7b3d7541ebfeae476ab86de0e593ba1f6a63ed4900e58f9fcc45d"} Nov 02 23:11:59.016915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:59.008918 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" podStartSLOduration=2.008879119 podStartE2EDuration="2.008879119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:11:59.005854104 +0000 UTC m=+1167.974385889" watchObservedRunningTime="2026-11-02 23:11:59.008879119 +0000 UTC m=+1167.977410903" Nov 02 23:11:59.532914 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:59.532252 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" Nov 02 23:11:59.540986 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:59.540076 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 02 23:11:59.540986 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:11:59.540136 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" podUID="73f3edf1-31bb-41a5-abb8-8a97ab98b1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 02 23:12:00.534300 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:00.534266 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 02 23:12:00.534783 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:00.534547 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" podUID="73f3edf1-31bb-41a5-abb8-8a97ab98b1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 02 23:12:01.534356 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:01.534309 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 02 23:12:01.534848 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:01.534375 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" podUID="73f3edf1-31bb-41a5-abb8-8a97ab98b1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 02 23:12:02.533419 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:02.533377 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 02 23:12:02.533749 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:02.533709 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" podUID="73f3edf1-31bb-41a5-abb8-8a97ab98b1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 02 23:12:03.532372 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:03.532330 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 02 23:12:03.533137 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:03.533100 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" podUID="73f3edf1-31bb-41a5-abb8-8a97ab98b1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 02 23:12:04.530878 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:04.530835 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 02 23:12:04.531097 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:04.530932 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" podUID="73f3edf1-31bb-41a5-abb8-8a97ab98b1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 02 23:12:06.382914 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:06.382043 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:12:06.382914 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:06.382128 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" podUID="3751da0e-99b6-4f2c-836e-699267f1428d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:06.461275 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:06.461234 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" Nov 02 23:12:06.488284 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:06.488251 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" Nov 02 23:12:06.548444 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:06.548400 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" Nov 02 23:12:09.884304 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:12:09.884260 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:12:12.341939 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:12.341908 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx"] Nov 02 23:12:12.347105 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:12.347082 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" Nov 02 23:12:12.415838 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:12.415806 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18c1e9a-d8c2-4438-8e96-2393552f495e-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx\" (UID: \"d18c1e9a-d8c2-4438-8e96-2393552f495e\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" Nov 02 23:12:12.416022 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:12.415845 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d18c1e9a-d8c2-4438-8e96-2393552f495e-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx\" (UID: \"d18c1e9a-d8c2-4438-8e96-2393552f495e\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" Nov 02 23:12:12.416022 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:12.415984 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d18c1e9a-d8c2-4438-8e96-2393552f495e-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx\" (UID: \"d18c1e9a-d8c2-4438-8e96-2393552f495e\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" Nov 02 23:12:12.416022 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:12.416010 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55xrf\" (UniqueName: \"kubernetes.io/projected/d18c1e9a-d8c2-4438-8e96-2393552f495e-kube-api-access-55xrf\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx\" (UID: \"d18c1e9a-d8c2-4438-8e96-2393552f495e\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" Nov 02 23:12:12.416181 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:12.416032 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d18c1e9a-d8c2-4438-8e96-2393552f495e-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx\" (UID: \"d18c1e9a-d8c2-4438-8e96-2393552f495e\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" Nov 02 23:12:12.416279 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:12.416257 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18c1e9a-d8c2-4438-8e96-2393552f495e-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx\" (UID: \"d18c1e9a-d8c2-4438-8e96-2393552f495e\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" Nov 02 23:12:12.416344 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:12.416313 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d18c1e9a-d8c2-4438-8e96-2393552f495e-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx\" (UID: \"d18c1e9a-d8c2-4438-8e96-2393552f495e\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" Nov 02 23:12:12.416398 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:12.416349 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d18c1e9a-d8c2-4438-8e96-2393552f495e-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx\" (UID: \"d18c1e9a-d8c2-4438-8e96-2393552f495e\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" Nov 02 23:12:12.416458 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:12.416425 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d18c1e9a-d8c2-4438-8e96-2393552f495e-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx\" (UID: \"d18c1e9a-d8c2-4438-8e96-2393552f495e\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" Nov 02 23:12:12.517397 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:12.517354 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d18c1e9a-d8c2-4438-8e96-2393552f495e-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx\" (UID: \"d18c1e9a-d8c2-4438-8e96-2393552f495e\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" Nov 02 23:12:12.517571 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:12.517464 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18c1e9a-d8c2-4438-8e96-2393552f495e-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx\" (UID: \"d18c1e9a-d8c2-4438-8e96-2393552f495e\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" Nov 02 23:12:12.517571 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:12.517491 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d18c1e9a-d8c2-4438-8e96-2393552f495e-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx\" (UID: \"d18c1e9a-d8c2-4438-8e96-2393552f495e\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" Nov 02 23:12:12.517571 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:12.517509 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d18c1e9a-d8c2-4438-8e96-2393552f495e-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx\" (UID: \"d18c1e9a-d8c2-4438-8e96-2393552f495e\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" Nov 02 23:12:12.517571 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:12.517541 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d18c1e9a-d8c2-4438-8e96-2393552f495e-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx\" (UID: \"d18c1e9a-d8c2-4438-8e96-2393552f495e\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" Nov 02 23:12:12.517737 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:12.517609 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18c1e9a-d8c2-4438-8e96-2393552f495e-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx\" (UID: \"d18c1e9a-d8c2-4438-8e96-2393552f495e\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" Nov 02 23:12:12.517737 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:12.517639 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d18c1e9a-d8c2-4438-8e96-2393552f495e-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx\" (UID: \"d18c1e9a-d8c2-4438-8e96-2393552f495e\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" Nov 02 23:12:12.517737 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:12.517704 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d18c1e9a-d8c2-4438-8e96-2393552f495e-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx\" (UID: \"d18c1e9a-d8c2-4438-8e96-2393552f495e\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" Nov 02 23:12:12.517737 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:12.517731 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55xrf\" (UniqueName: \"kubernetes.io/projected/d18c1e9a-d8c2-4438-8e96-2393552f495e-kube-api-access-55xrf\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx\" (UID: \"d18c1e9a-d8c2-4438-8e96-2393552f495e\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" Nov 02 23:12:12.517943 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:12.517916 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d18c1e9a-d8c2-4438-8e96-2393552f495e-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx\" (UID: \"d18c1e9a-d8c2-4438-8e96-2393552f495e\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" Nov 02 23:12:12.518186 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:12.518161 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18c1e9a-d8c2-4438-8e96-2393552f495e-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx\" (UID: \"d18c1e9a-d8c2-4438-8e96-2393552f495e\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" Nov 02 23:12:12.518313 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:12.518204 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d18c1e9a-d8c2-4438-8e96-2393552f495e-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx\" (UID: \"d18c1e9a-d8c2-4438-8e96-2393552f495e\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" Nov 02 23:12:12.518313 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:12.518229 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18c1e9a-d8c2-4438-8e96-2393552f495e-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx\" (UID: \"d18c1e9a-d8c2-4438-8e96-2393552f495e\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" Nov 02 23:12:12.518313 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:12.518236 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d18c1e9a-d8c2-4438-8e96-2393552f495e-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx\" (UID: \"d18c1e9a-d8c2-4438-8e96-2393552f495e\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" Nov 02 23:12:12.519961 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:12.519940 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d18c1e9a-d8c2-4438-8e96-2393552f495e-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx\" (UID: \"d18c1e9a-d8c2-4438-8e96-2393552f495e\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" Nov 02 23:12:12.520176 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:12.520160 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d18c1e9a-d8c2-4438-8e96-2393552f495e-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx\" (UID: \"d18c1e9a-d8c2-4438-8e96-2393552f495e\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" Nov 02 23:12:12.533709 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:12.533686 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx"] Nov 02 23:12:12.576671 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:12.576636 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d18c1e9a-d8c2-4438-8e96-2393552f495e-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx\" (UID: \"d18c1e9a-d8c2-4438-8e96-2393552f495e\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" Nov 02 23:12:12.576837 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:12.576815 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55xrf\" (UniqueName: \"kubernetes.io/projected/d18c1e9a-d8c2-4438-8e96-2393552f495e-kube-api-access-55xrf\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx\" (UID: \"d18c1e9a-d8c2-4438-8e96-2393552f495e\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" Nov 02 23:12:12.661772 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:12.661678 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" Nov 02 23:12:13.076409 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:13.076371 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx"] Nov 02 23:12:13.112561 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:12:13.112529 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd18c1e9a_d8c2_4438_8e96_2393552f495e.slice/crio-73ae02e5fb6f93ef56b1a9d57b8990fc5dd88bd1f45f31110795059770526c4a WatchSource:0}: Error finding container 73ae02e5fb6f93ef56b1a9d57b8990fc5dd88bd1f45f31110795059770526c4a: Status 404 returned error can't find the container with id 73ae02e5fb6f93ef56b1a9d57b8990fc5dd88bd1f45f31110795059770526c4a Nov 02 23:12:13.114959 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:13.114924 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:12:13.115042 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:13.114999 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:12:13.115042 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:13.115033 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:12:14.051556 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:14.051518 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" event={"ID":"d18c1e9a-d8c2-4438-8e96-2393552f495e","Type":"ContainerStarted","Data":"9e43b7e56f165e0d0165337b511036ea62356c7a748236f22035f656db077dc2"} Nov 02 23:12:14.052020 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:14.051570 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" event={"ID":"d18c1e9a-d8c2-4438-8e96-2393552f495e","Type":"ContainerStarted","Data":"73ae02e5fb6f93ef56b1a9d57b8990fc5dd88bd1f45f31110795059770526c4a"} Nov 02 23:12:14.281589 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:14.281535 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" podStartSLOduration=2.281519242 podStartE2EDuration="2.281519242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:12:14.232415517 +0000 UTC m=+1183.200947302" watchObservedRunningTime="2026-11-02 23:12:14.281519242 +0000 UTC m=+1183.250051086" Nov 02 23:12:14.662772 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:14.662735 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" Nov 02 23:12:14.664548 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:14.664520 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 02 23:12:14.664678 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:14.664579 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" podUID="d18c1e9a-d8c2-4438-8e96-2393552f495e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 02 23:12:15.662117 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:15.662078 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 02 23:12:15.662513 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:15.662137 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" podUID="d18c1e9a-d8c2-4438-8e96-2393552f495e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 02 23:12:16.662903 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:16.662852 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 02 23:12:16.663332 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:16.662939 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" podUID="d18c1e9a-d8c2-4438-8e96-2393552f495e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 02 23:12:17.662862 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:17.662824 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 02 23:12:17.663090 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:17.662877 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" podUID="d18c1e9a-d8c2-4438-8e96-2393552f495e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 02 23:12:18.662301 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:18.662266 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 02 23:12:18.662482 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:18.662317 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" podUID="d18c1e9a-d8c2-4438-8e96-2393552f495e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 02 23:12:19.667925 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:19.667358 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 02 23:12:19.667925 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:19.667423 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" podUID="d18c1e9a-d8c2-4438-8e96-2393552f495e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 02 23:12:20.667626 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:20.667592 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" Nov 02 23:12:20.667929 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:20.667882 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" Nov 02 23:12:20.668703 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:20.668682 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" Nov 02 23:12:26.886368 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:26.886310 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k"] Nov 02 23:12:26.910860 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:26.910828 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" Nov 02 23:12:26.962923 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:26.948826 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k"] Nov 02 23:12:27.054049 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.053008 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7"] Nov 02 23:12:27.058311 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.058216 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" Nov 02 23:12:27.092938 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.091631 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-workload-certs\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k\" (UID: \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" Nov 02 23:12:27.092938 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.091728 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-istio-envoy\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k\" (UID: \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" Nov 02 23:12:27.092938 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.091771 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k6fb\" (UniqueName: \"kubernetes.io/projected/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-kube-api-access-4k6fb\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k\" (UID: \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" Nov 02 23:12:27.092938 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.091871 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-istio-data\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k\" (UID: \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" Nov 02 23:12:27.092938 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.091952 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-istio-token\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k\" (UID: \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" Nov 02 23:12:27.092938 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.091979 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-istiod-ca-cert\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k\" (UID: \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" Nov 02 23:12:27.092938 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.092029 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-istio-podinfo\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k\" (UID: \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" Nov 02 23:12:27.092938 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.092062 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-credential-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k\" (UID: \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" Nov 02 23:12:27.092938 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.092146 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-workload-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k\" (UID: \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" Nov 02 23:12:27.190206 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.111282 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-fb6e1d\"/\"distinct-2-w4-fb6e1d-kserve-self-signed-certs\"" Nov 02 23:12:27.231539 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.231503 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-workload-certs\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k\" (UID: \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" Nov 02 23:12:27.279326 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.279258 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf2b01c6-8001-49e3-908a-38cffb88edfa-home\") pod \"distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7\" (UID: \"cf2b01c6-8001-49e3-908a-38cffb88edfa\") " pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" Nov 02 23:12:27.279845 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.279815 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-istio-envoy\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k\" (UID: \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" Nov 02 23:12:27.280124 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.280089 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k6fb\" (UniqueName: \"kubernetes.io/projected/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-kube-api-access-4k6fb\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k\" (UID: \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" Nov 02 23:12:27.285371 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.285336 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf2b01c6-8001-49e3-908a-38cffb88edfa-model-cache\") pod \"distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7\" (UID: \"cf2b01c6-8001-49e3-908a-38cffb88edfa\") " pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" Nov 02 23:12:27.285505 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.285399 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwdsg\" (UniqueName: \"kubernetes.io/projected/cf2b01c6-8001-49e3-908a-38cffb88edfa-kube-api-access-pwdsg\") pod \"distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7\" (UID: \"cf2b01c6-8001-49e3-908a-38cffb88edfa\") " pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" Nov 02 23:12:27.285505 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.285447 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-istio-data\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k\" (UID: \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" Nov 02 23:12:27.285634 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.285564 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-istio-token\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k\" (UID: \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" Nov 02 23:12:27.285634 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.285594 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-istiod-ca-cert\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k\" (UID: \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" Nov 02 23:12:27.285634 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.285618 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf2b01c6-8001-49e3-908a-38cffb88edfa-tls-certs\") pod \"distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7\" (UID: \"cf2b01c6-8001-49e3-908a-38cffb88edfa\") " pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" Nov 02 23:12:27.285810 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.285680 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf2b01c6-8001-49e3-908a-38cffb88edfa-dshm\") pod \"distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7\" (UID: \"cf2b01c6-8001-49e3-908a-38cffb88edfa\") " pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" Nov 02 23:12:27.285810 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.285710 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-istio-podinfo\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k\" (UID: \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" Nov 02 23:12:27.285810 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.285745 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-credential-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k\" (UID: \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" Nov 02 23:12:27.285986 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.285816 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf2b01c6-8001-49e3-908a-38cffb88edfa-tmp-dir\") pod \"distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7\" (UID: \"cf2b01c6-8001-49e3-908a-38cffb88edfa\") " pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" Nov 02 23:12:27.285986 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.285911 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-workload-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k\" (UID: \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" Nov 02 23:12:27.344937 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.339019 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-istiod-ca-cert\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k\" (UID: \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" Nov 02 23:12:27.365951 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.244735 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-workload-certs\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k\" (UID: \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" Nov 02 23:12:27.435247 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.435205 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf2b01c6-8001-49e3-908a-38cffb88edfa-home\") pod \"distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7\" (UID: \"cf2b01c6-8001-49e3-908a-38cffb88edfa\") " pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" Nov 02 23:12:27.435609 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.435584 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf2b01c6-8001-49e3-908a-38cffb88edfa-model-cache\") pod \"distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7\" (UID: \"cf2b01c6-8001-49e3-908a-38cffb88edfa\") " pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" Nov 02 23:12:27.435773 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.435755 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwdsg\" (UniqueName: \"kubernetes.io/projected/cf2b01c6-8001-49e3-908a-38cffb88edfa-kube-api-access-pwdsg\") pod \"distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7\" (UID: \"cf2b01c6-8001-49e3-908a-38cffb88edfa\") " pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" Nov 02 23:12:27.435984 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.435965 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf2b01c6-8001-49e3-908a-38cffb88edfa-tls-certs\") pod \"distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7\" (UID: \"cf2b01c6-8001-49e3-908a-38cffb88edfa\") " pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" Nov 02 23:12:27.436156 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.436140 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf2b01c6-8001-49e3-908a-38cffb88edfa-dshm\") pod \"distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7\" (UID: \"cf2b01c6-8001-49e3-908a-38cffb88edfa\") " pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" Nov 02 23:12:27.461165 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.461065 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-credential-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k\" (UID: \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" Nov 02 23:12:27.465685 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.465656 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-workload-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k\" (UID: \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" Nov 02 23:12:27.472921 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.466104 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-istio-data\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k\" (UID: \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" Nov 02 23:12:27.506934 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.474176 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf2b01c6-8001-49e3-908a-38cffb88edfa-home\") pod \"distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7\" (UID: \"cf2b01c6-8001-49e3-908a-38cffb88edfa\") " pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" Nov 02 23:12:27.610672 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.507702 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf2b01c6-8001-49e3-908a-38cffb88edfa-model-cache\") pod \"distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7\" (UID: \"cf2b01c6-8001-49e3-908a-38cffb88edfa\") " pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" Nov 02 23:12:27.611343 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.611303 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-istio-envoy\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k\" (UID: \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" Nov 02 23:12:27.616929 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.614755 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf2b01c6-8001-49e3-908a-38cffb88edfa-dshm\") pod \"distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7\" (UID: \"cf2b01c6-8001-49e3-908a-38cffb88edfa\") " pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" Nov 02 23:12:27.616929 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.615541 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf2b01c6-8001-49e3-908a-38cffb88edfa-tmp-dir\") pod \"distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7\" (UID: \"cf2b01c6-8001-49e3-908a-38cffb88edfa\") " pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" Nov 02 23:12:27.645464 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.645408 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf2b01c6-8001-49e3-908a-38cffb88edfa-tmp-dir\") pod \"distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7\" (UID: \"cf2b01c6-8001-49e3-908a-38cffb88edfa\") " pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" Nov 02 23:12:27.664911 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.646206 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-istio-podinfo\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k\" (UID: \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" Nov 02 23:12:27.665349 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.654455 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k6fb\" (UniqueName: \"kubernetes.io/projected/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-kube-api-access-4k6fb\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k\" (UID: \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" Nov 02 23:12:27.665505 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.657375 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-istio-token\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k\" (UID: \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" Nov 02 23:12:27.665641 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.661731 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf2b01c6-8001-49e3-908a-38cffb88edfa-tls-certs\") pod \"distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7\" (UID: \"cf2b01c6-8001-49e3-908a-38cffb88edfa\") " pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" Nov 02 23:12:27.665752 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.662458 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwdsg\" (UniqueName: \"kubernetes.io/projected/cf2b01c6-8001-49e3-908a-38cffb88edfa-kube-api-access-pwdsg\") pod \"distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7\" (UID: \"cf2b01c6-8001-49e3-908a-38cffb88edfa\") " pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" Nov 02 23:12:27.812296 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.730746 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7"] Nov 02 23:12:27.835579 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.835534 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" Nov 02 23:12:27.850152 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:27.850117 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" Nov 02 23:12:28.748949 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:28.747829 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp"] Nov 02 23:12:28.820983 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:28.818294 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" Nov 02 23:12:28.830192 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:28.822174 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk"] Nov 02 23:12:28.905684 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:28.902685 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0450c8f1-c6f6-4d65-a0a0-7e954133793d-credential-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp\" (UID: \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" Nov 02 23:12:28.905684 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:28.902739 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0450c8f1-c6f6-4d65-a0a0-7e954133793d-istio-envoy\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp\" (UID: \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" Nov 02 23:12:28.905684 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:28.902797 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0450c8f1-c6f6-4d65-a0a0-7e954133793d-istio-data\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp\" (UID: \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" Nov 02 23:12:28.905684 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:28.902842 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0450c8f1-c6f6-4d65-a0a0-7e954133793d-workload-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp\" (UID: \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" Nov 02 23:12:28.905684 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:28.902931 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0450c8f1-c6f6-4d65-a0a0-7e954133793d-istiod-ca-cert\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp\" (UID: \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" Nov 02 23:12:28.905684 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:28.903045 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkprc\" (UniqueName: \"kubernetes.io/projected/0450c8f1-c6f6-4d65-a0a0-7e954133793d-kube-api-access-zkprc\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp\" (UID: \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" Nov 02 23:12:28.905684 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:28.903085 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0450c8f1-c6f6-4d65-a0a0-7e954133793d-workload-certs\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp\" (UID: \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" Nov 02 23:12:28.905684 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:28.903119 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0450c8f1-c6f6-4d65-a0a0-7e954133793d-istio-token\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp\" (UID: \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" Nov 02 23:12:28.905684 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:28.903190 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0450c8f1-c6f6-4d65-a0a0-7e954133793d-istio-podinfo\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp\" (UID: \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" Nov 02 23:12:29.138540 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.137489 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkprc\" (UniqueName: \"kubernetes.io/projected/0450c8f1-c6f6-4d65-a0a0-7e954133793d-kube-api-access-zkprc\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp\" (UID: \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" Nov 02 23:12:29.138540 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.137551 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0450c8f1-c6f6-4d65-a0a0-7e954133793d-workload-certs\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp\" (UID: \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" Nov 02 23:12:29.138540 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.137577 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0450c8f1-c6f6-4d65-a0a0-7e954133793d-istio-token\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp\" (UID: \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" Nov 02 23:12:29.138540 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.137636 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0450c8f1-c6f6-4d65-a0a0-7e954133793d-istio-podinfo\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp\" (UID: \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" Nov 02 23:12:29.138540 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.137847 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0450c8f1-c6f6-4d65-a0a0-7e954133793d-credential-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp\" (UID: \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" Nov 02 23:12:29.138540 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.137880 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0450c8f1-c6f6-4d65-a0a0-7e954133793d-istio-envoy\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp\" (UID: \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" Nov 02 23:12:29.138540 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.137950 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0450c8f1-c6f6-4d65-a0a0-7e954133793d-istio-data\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp\" (UID: \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" Nov 02 23:12:29.138540 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.137994 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0450c8f1-c6f6-4d65-a0a0-7e954133793d-workload-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp\" (UID: \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" Nov 02 23:12:29.138540 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.138056 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0450c8f1-c6f6-4d65-a0a0-7e954133793d-istiod-ca-cert\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp\" (UID: \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" Nov 02 23:12:29.141950 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.141679 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0450c8f1-c6f6-4d65-a0a0-7e954133793d-workload-certs\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp\" (UID: \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" Nov 02 23:12:29.146291 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.145624 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0450c8f1-c6f6-4d65-a0a0-7e954133793d-workload-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp\" (UID: \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" Nov 02 23:12:29.148964 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.147162 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0450c8f1-c6f6-4d65-a0a0-7e954133793d-istiod-ca-cert\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp\" (UID: \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" Nov 02 23:12:29.149377 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.149354 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0450c8f1-c6f6-4d65-a0a0-7e954133793d-credential-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp\" (UID: \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" Nov 02 23:12:29.163134 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.163103 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0450c8f1-c6f6-4d65-a0a0-7e954133793d-istio-data\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp\" (UID: \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" Nov 02 23:12:29.173218 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.173186 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" Nov 02 23:12:29.209830 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.197395 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0450c8f1-c6f6-4d65-a0a0-7e954133793d-istio-podinfo\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp\" (UID: \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" Nov 02 23:12:29.209830 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.205640 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0450c8f1-c6f6-4d65-a0a0-7e954133793d-istio-token\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp\" (UID: \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" Nov 02 23:12:29.215309 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.215278 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp"] Nov 02 23:12:29.215470 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.215344 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk"] Nov 02 23:12:29.223073 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.222998 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0450c8f1-c6f6-4d65-a0a0-7e954133793d-istio-envoy\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp\" (UID: \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" Nov 02 23:12:29.225994 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.225938 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkprc\" (UniqueName: \"kubernetes.io/projected/0450c8f1-c6f6-4d65-a0a0-7e954133793d-kube-api-access-zkprc\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp\" (UID: \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" Nov 02 23:12:29.243260 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.243172 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca62790c-9f83-45df-9bb8-6360b271dab0-workload-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk\" (UID: \"ca62790c-9f83-45df-9bb8-6360b271dab0\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" Nov 02 23:12:29.243427 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.243288 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca62790c-9f83-45df-9bb8-6360b271dab0-istio-token\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk\" (UID: \"ca62790c-9f83-45df-9bb8-6360b271dab0\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" Nov 02 23:12:29.243427 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.243329 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ln22\" (UniqueName: \"kubernetes.io/projected/ca62790c-9f83-45df-9bb8-6360b271dab0-kube-api-access-8ln22\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk\" (UID: \"ca62790c-9f83-45df-9bb8-6360b271dab0\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" Nov 02 23:12:29.243427 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.243357 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca62790c-9f83-45df-9bb8-6360b271dab0-credential-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk\" (UID: \"ca62790c-9f83-45df-9bb8-6360b271dab0\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" Nov 02 23:12:29.243427 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.243379 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca62790c-9f83-45df-9bb8-6360b271dab0-istio-podinfo\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk\" (UID: \"ca62790c-9f83-45df-9bb8-6360b271dab0\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" Nov 02 23:12:29.243653 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.243430 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca62790c-9f83-45df-9bb8-6360b271dab0-istiod-ca-cert\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk\" (UID: \"ca62790c-9f83-45df-9bb8-6360b271dab0\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" Nov 02 23:12:29.243653 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.243466 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca62790c-9f83-45df-9bb8-6360b271dab0-istio-data\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk\" (UID: \"ca62790c-9f83-45df-9bb8-6360b271dab0\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" Nov 02 23:12:29.243653 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.243579 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca62790c-9f83-45df-9bb8-6360b271dab0-istio-envoy\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk\" (UID: \"ca62790c-9f83-45df-9bb8-6360b271dab0\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" Nov 02 23:12:29.244134 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.244110 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca62790c-9f83-45df-9bb8-6360b271dab0-workload-certs\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk\" (UID: \"ca62790c-9f83-45df-9bb8-6360b271dab0\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" Nov 02 23:12:29.345240 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.345156 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca62790c-9f83-45df-9bb8-6360b271dab0-istio-token\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk\" (UID: \"ca62790c-9f83-45df-9bb8-6360b271dab0\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" Nov 02 23:12:29.345240 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.345198 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ln22\" (UniqueName: \"kubernetes.io/projected/ca62790c-9f83-45df-9bb8-6360b271dab0-kube-api-access-8ln22\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk\" (UID: \"ca62790c-9f83-45df-9bb8-6360b271dab0\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" Nov 02 23:12:29.345240 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.345218 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca62790c-9f83-45df-9bb8-6360b271dab0-credential-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk\" (UID: \"ca62790c-9f83-45df-9bb8-6360b271dab0\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" Nov 02 23:12:29.345240 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.345233 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca62790c-9f83-45df-9bb8-6360b271dab0-istio-podinfo\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk\" (UID: \"ca62790c-9f83-45df-9bb8-6360b271dab0\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" Nov 02 23:12:29.345586 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.345270 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca62790c-9f83-45df-9bb8-6360b271dab0-istiod-ca-cert\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk\" (UID: \"ca62790c-9f83-45df-9bb8-6360b271dab0\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" Nov 02 23:12:29.345586 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.345295 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca62790c-9f83-45df-9bb8-6360b271dab0-istio-data\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk\" (UID: \"ca62790c-9f83-45df-9bb8-6360b271dab0\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" Nov 02 23:12:29.345586 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.345369 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca62790c-9f83-45df-9bb8-6360b271dab0-istio-envoy\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk\" (UID: \"ca62790c-9f83-45df-9bb8-6360b271dab0\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" Nov 02 23:12:29.345586 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.345446 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca62790c-9f83-45df-9bb8-6360b271dab0-workload-certs\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk\" (UID: \"ca62790c-9f83-45df-9bb8-6360b271dab0\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" Nov 02 23:12:29.345586 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.345503 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca62790c-9f83-45df-9bb8-6360b271dab0-workload-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk\" (UID: \"ca62790c-9f83-45df-9bb8-6360b271dab0\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" Nov 02 23:12:29.345819 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.345660 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca62790c-9f83-45df-9bb8-6360b271dab0-credential-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk\" (UID: \"ca62790c-9f83-45df-9bb8-6360b271dab0\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" Nov 02 23:12:29.345819 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.345752 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca62790c-9f83-45df-9bb8-6360b271dab0-istio-data\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk\" (UID: \"ca62790c-9f83-45df-9bb8-6360b271dab0\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" Nov 02 23:12:29.345926 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.345815 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca62790c-9f83-45df-9bb8-6360b271dab0-workload-certs\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk\" (UID: \"ca62790c-9f83-45df-9bb8-6360b271dab0\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" Nov 02 23:12:29.345926 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.345878 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca62790c-9f83-45df-9bb8-6360b271dab0-workload-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk\" (UID: \"ca62790c-9f83-45df-9bb8-6360b271dab0\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" Nov 02 23:12:29.346030 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.346018 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca62790c-9f83-45df-9bb8-6360b271dab0-istiod-ca-cert\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk\" (UID: \"ca62790c-9f83-45df-9bb8-6360b271dab0\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" Nov 02 23:12:29.347683 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.347662 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca62790c-9f83-45df-9bb8-6360b271dab0-istio-envoy\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk\" (UID: \"ca62790c-9f83-45df-9bb8-6360b271dab0\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" Nov 02 23:12:29.348058 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.348039 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca62790c-9f83-45df-9bb8-6360b271dab0-istio-podinfo\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk\" (UID: \"ca62790c-9f83-45df-9bb8-6360b271dab0\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" Nov 02 23:12:29.396659 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.396628 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7"] Nov 02 23:12:29.397315 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:12:29.397285 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf2b01c6_8001_49e3_908a_38cffb88edfa.slice/crio-50f618a8a037feb8fca96822140adea3ff1a08c4549bbe8f7c77f4b907b5da8b WatchSource:0}: Error finding container 50f618a8a037feb8fca96822140adea3ff1a08c4549bbe8f7c77f4b907b5da8b: Status 404 returned error can't find the container with id 50f618a8a037feb8fca96822140adea3ff1a08c4549bbe8f7c77f4b907b5da8b Nov 02 23:12:29.405503 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.405474 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca62790c-9f83-45df-9bb8-6360b271dab0-istio-token\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk\" (UID: \"ca62790c-9f83-45df-9bb8-6360b271dab0\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" Nov 02 23:12:29.405503 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.405488 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ln22\" (UniqueName: \"kubernetes.io/projected/ca62790c-9f83-45df-9bb8-6360b271dab0-kube-api-access-8ln22\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk\" (UID: \"ca62790c-9f83-45df-9bb8-6360b271dab0\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" Nov 02 23:12:29.469764 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.469733 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" Nov 02 23:12:29.473529 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.473502 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k"] Nov 02 23:12:29.475297 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:12:29.475272 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cfe1803_0eec_4193_aa8f_05b3699a0ed2.slice/crio-dac71ce00b193f064b47ec110ce7f4c2e1c8c0b00fb731172d20516652bb9566 WatchSource:0}: Error finding container dac71ce00b193f064b47ec110ce7f4c2e1c8c0b00fb731172d20516652bb9566: Status 404 returned error can't find the container with id dac71ce00b193f064b47ec110ce7f4c2e1c8c0b00fb731172d20516652bb9566 Nov 02 23:12:29.483066 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.483022 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:12:29.483426 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.483395 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:12:29.483597 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.483477 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:12:29.586942 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.577589 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" Nov 02 23:12:29.927926 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.927791 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" event={"ID":"cf2b01c6-8001-49e3-908a-38cffb88edfa","Type":"ContainerStarted","Data":"161090faabb34d26de936b2a96fe2e49a0fbb1cf60a50ca1b8442b78ec1ae00a"} Nov 02 23:12:29.927926 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.927844 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" event={"ID":"cf2b01c6-8001-49e3-908a-38cffb88edfa","Type":"ContainerStarted","Data":"50f618a8a037feb8fca96822140adea3ff1a08c4549bbe8f7c77f4b907b5da8b"} Nov 02 23:12:29.929957 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.929924 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" event={"ID":"2cfe1803-0eec-4193-aa8f-05b3699a0ed2","Type":"ContainerStarted","Data":"2898c23dab897cb166e21b4a5c1a3aac6a2533aa31db6c41b8f3d1762c679314"} Nov 02 23:12:29.930081 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.929971 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" event={"ID":"2cfe1803-0eec-4193-aa8f-05b3699a0ed2","Type":"ContainerStarted","Data":"dac71ce00b193f064b47ec110ce7f4c2e1c8c0b00fb731172d20516652bb9566"} Nov 02 23:12:29.963522 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.963481 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp"] Nov 02 23:12:29.964037 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:12:29.964012 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0450c8f1_c6f6_4d65_a0a0_7e954133793d.slice/crio-05fe7095208864455dde002b63539781dacfb4e1b47ef90ec4cc50eb011b0386 WatchSource:0}: Error finding container 05fe7095208864455dde002b63539781dacfb4e1b47ef90ec4cc50eb011b0386: Status 404 returned error can't find the container with id 05fe7095208864455dde002b63539781dacfb4e1b47ef90ec4cc50eb011b0386 Nov 02 23:12:29.966485 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.966455 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:12:29.966562 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.966518 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:12:29.966562 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:29.966547 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:12:30.009606 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:12:30.009558 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca62790c_9f83_45df_9bb8_6360b271dab0.slice/crio-7a28035e8eb7f2df9854bf06026b9df25e9da4f66b4914a9489410199048f8db WatchSource:0}: Error finding container 7a28035e8eb7f2df9854bf06026b9df25e9da4f66b4914a9489410199048f8db: Status 404 returned error can't find the container with id 7a28035e8eb7f2df9854bf06026b9df25e9da4f66b4914a9489410199048f8db Nov 02 23:12:30.010054 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:30.010029 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk"] Nov 02 23:12:30.013065 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:30.013026 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:12:30.013149 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:30.013124 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:12:30.013184 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:30.013172 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:12:30.037216 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:30.037167 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" podStartSLOduration=4.037153351 podStartE2EDuration="4.037153351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:12:30.036452823 +0000 UTC m=+1199.004984609" watchObservedRunningTime="2026-11-02 23:12:30.037153351 +0000 UTC m=+1199.005685134" Nov 02 23:12:30.187962 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:30.185658 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" podStartSLOduration=4.185635728 podStartE2EDuration="4.185635728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:12:30.171304924 +0000 UTC m=+1199.139836710" watchObservedRunningTime="2026-11-02 23:12:30.185635728 +0000 UTC m=+1199.154167512" Nov 02 23:12:30.836776 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:30.836736 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" Nov 02 23:12:30.850921 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:30.850874 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" Nov 02 23:12:30.852409 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:30.852375 5076 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 02 23:12:30.852556 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:30.852427 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" podUID="2cfe1803-0eec-4193-aa8f-05b3699a0ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 02 23:12:30.959907 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:30.959856 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" event={"ID":"ca62790c-9f83-45df-9bb8-6360b271dab0","Type":"ContainerStarted","Data":"6c99bd4e46ace22a9a0c773655251c1621120c4d6925fe3892092c2378c731f1"} Nov 02 23:12:30.960390 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:30.959982 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" event={"ID":"ca62790c-9f83-45df-9bb8-6360b271dab0","Type":"ContainerStarted","Data":"7a28035e8eb7f2df9854bf06026b9df25e9da4f66b4914a9489410199048f8db"} Nov 02 23:12:30.964842 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:30.964797 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" event={"ID":"0450c8f1-c6f6-4d65-a0a0-7e954133793d","Type":"ContainerStarted","Data":"0aef3170f0e567d4efbac8e08eb72722c8b739ce06e6d888866cd0e04475081f"} Nov 02 23:12:30.965817 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:30.965791 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" event={"ID":"0450c8f1-c6f6-4d65-a0a0-7e954133793d","Type":"ContainerStarted","Data":"05fe7095208864455dde002b63539781dacfb4e1b47ef90ec4cc50eb011b0386"} Nov 02 23:12:31.137965 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:31.137063 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" podStartSLOduration=3.1370434 podStartE2EDuration="3.1370434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:12:31.1345137 +0000 UTC m=+1200.103045486" watchObservedRunningTime="2026-11-02 23:12:31.1370434 +0000 UTC m=+1200.105575180" Nov 02 23:12:31.478914 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:31.474731 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" Nov 02 23:12:31.496915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:31.493268 5076 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 02 23:12:31.496915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:31.493365 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" podUID="0450c8f1-c6f6-4d65-a0a0-7e954133793d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 02 23:12:31.530915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:31.529993 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 02 23:12:31.554067 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:31.531335 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 02 23:12:31.589916 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:31.580050 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" Nov 02 23:12:31.590098 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:31.590055 5076 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 02 23:12:31.590179 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:31.590148 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" podUID="ca62790c-9f83-45df-9bb8-6360b271dab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 02 23:12:31.872000 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:31.869699 5076 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 02 23:12:31.872000 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:31.869773 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" podUID="2cfe1803-0eec-4193-aa8f-05b3699a0ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 02 23:12:32.471354 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:32.471310 5076 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 02 23:12:32.471978 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:32.471938 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" podUID="0450c8f1-c6f6-4d65-a0a0-7e954133793d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 02 23:12:32.580355 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:32.580317 5076 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 02 23:12:32.580637 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:32.580603 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" podUID="ca62790c-9f83-45df-9bb8-6360b271dab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 02 23:12:32.868976 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:32.865061 5076 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 02 23:12:32.868976 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:32.865120 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" podUID="2cfe1803-0eec-4193-aa8f-05b3699a0ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 02 23:12:33.470355 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:33.470315 5076 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 02 23:12:33.470606 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:33.470580 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" podUID="0450c8f1-c6f6-4d65-a0a0-7e954133793d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 02 23:12:33.580362 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:33.580323 5076 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 02 23:12:33.581014 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:33.580978 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" podUID="ca62790c-9f83-45df-9bb8-6360b271dab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 02 23:12:33.852181 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:33.852097 5076 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 02 23:12:33.852426 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:33.852399 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" podUID="2cfe1803-0eec-4193-aa8f-05b3699a0ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 02 23:12:34.475369 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:34.475322 5076 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 02 23:12:34.475793 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:34.475762 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" podUID="0450c8f1-c6f6-4d65-a0a0-7e954133793d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 02 23:12:34.583556 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:34.583521 5076 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 02 23:12:34.584233 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:34.584201 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" podUID="ca62790c-9f83-45df-9bb8-6360b271dab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 02 23:12:34.858802 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:34.858711 5076 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 02 23:12:34.859091 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:34.859059 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" podUID="2cfe1803-0eec-4193-aa8f-05b3699a0ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 02 23:12:35.470916 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:35.470255 5076 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 02 23:12:35.470916 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:35.470317 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" podUID="0450c8f1-c6f6-4d65-a0a0-7e954133793d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 02 23:12:35.585943 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:35.584015 5076 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 02 23:12:35.585943 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:35.584080 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" podUID="ca62790c-9f83-45df-9bb8-6360b271dab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 02 23:12:35.858002 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:35.857253 5076 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 02 23:12:35.858002 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:35.857324 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" podUID="2cfe1803-0eec-4193-aa8f-05b3699a0ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 02 23:12:36.470386 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:36.470340 5076 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 02 23:12:36.470613 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:36.470410 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" podUID="0450c8f1-c6f6-4d65-a0a0-7e954133793d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 02 23:12:36.580937 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:36.578604 5076 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 02 23:12:36.580937 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:36.578670 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" podUID="ca62790c-9f83-45df-9bb8-6360b271dab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 02 23:12:36.851880 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:36.851786 5076 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 02 23:12:36.851880 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:36.851851 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" podUID="2cfe1803-0eec-4193-aa8f-05b3699a0ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 02 23:12:37.471153 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:37.471112 5076 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 02 23:12:37.471359 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:37.471176 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" podUID="0450c8f1-c6f6-4d65-a0a0-7e954133793d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 02 23:12:37.836353 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:37.836265 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" Nov 02 23:12:37.849614 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:37.849587 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" Nov 02 23:12:37.855712 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:37.855685 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" Nov 02 23:12:37.856203 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:37.856086 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" Nov 02 23:12:37.856985 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:37.856966 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" Nov 02 23:12:37.959061 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:37.959001 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" podStartSLOduration=9.958981872 podStartE2EDuration="9.958981872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:12:31.26074565 +0000 UTC m=+1200.229277435" watchObservedRunningTime="2026-11-02 23:12:37.958981872 +0000 UTC m=+1206.927513656" Nov 02 23:12:38.143946 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:38.143823 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" Nov 02 23:12:38.147595 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:38.146055 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" Nov 02 23:12:38.147595 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:38.146534 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" Nov 02 23:12:38.147595 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:38.147507 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" Nov 02 23:12:38.474841 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:38.474807 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" Nov 02 23:12:38.475357 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:38.475335 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" Nov 02 23:12:38.475975 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:38.475956 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" Nov 02 23:12:39.883879 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:12:39.883845 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:12:41.964342 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:41.964264 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl"] Nov 02 23:12:41.968196 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:41.968178 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" Nov 02 23:12:42.015715 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:42.015680 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c34dfac-f28d-420f-a6da-8279cae95313-istio-envoy\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl\" (UID: \"3c34dfac-f28d-420f-a6da-8279cae95313\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" Nov 02 23:12:42.015884 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:42.015722 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c34dfac-f28d-420f-a6da-8279cae95313-workload-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl\" (UID: \"3c34dfac-f28d-420f-a6da-8279cae95313\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" Nov 02 23:12:42.015884 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:42.015865 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkgsw\" (UniqueName: \"kubernetes.io/projected/3c34dfac-f28d-420f-a6da-8279cae95313-kube-api-access-mkgsw\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl\" (UID: \"3c34dfac-f28d-420f-a6da-8279cae95313\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" Nov 02 23:12:42.016002 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:42.015972 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c34dfac-f28d-420f-a6da-8279cae95313-credential-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl\" (UID: \"3c34dfac-f28d-420f-a6da-8279cae95313\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" Nov 02 23:12:42.016120 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:42.016102 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c34dfac-f28d-420f-a6da-8279cae95313-istio-token\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl\" (UID: \"3c34dfac-f28d-420f-a6da-8279cae95313\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" Nov 02 23:12:42.016190 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:42.016141 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c34dfac-f28d-420f-a6da-8279cae95313-istiod-ca-cert\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl\" (UID: \"3c34dfac-f28d-420f-a6da-8279cae95313\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" Nov 02 23:12:42.016258 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:42.016240 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c34dfac-f28d-420f-a6da-8279cae95313-istio-data\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl\" (UID: \"3c34dfac-f28d-420f-a6da-8279cae95313\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" Nov 02 23:12:42.016333 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:42.016314 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c34dfac-f28d-420f-a6da-8279cae95313-istio-podinfo\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl\" (UID: \"3c34dfac-f28d-420f-a6da-8279cae95313\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" Nov 02 23:12:42.016464 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:42.016356 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c34dfac-f28d-420f-a6da-8279cae95313-workload-certs\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl\" (UID: \"3c34dfac-f28d-420f-a6da-8279cae95313\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" Nov 02 23:12:42.071905 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:42.071853 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl"] Nov 02 23:12:42.117111 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:42.117077 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c34dfac-f28d-420f-a6da-8279cae95313-credential-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl\" (UID: \"3c34dfac-f28d-420f-a6da-8279cae95313\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" Nov 02 23:12:42.117271 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:42.117233 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c34dfac-f28d-420f-a6da-8279cae95313-istio-token\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl\" (UID: \"3c34dfac-f28d-420f-a6da-8279cae95313\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" Nov 02 23:12:42.117359 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:42.117268 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c34dfac-f28d-420f-a6da-8279cae95313-istiod-ca-cert\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl\" (UID: \"3c34dfac-f28d-420f-a6da-8279cae95313\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" Nov 02 23:12:42.117359 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:42.117311 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c34dfac-f28d-420f-a6da-8279cae95313-istio-data\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl\" (UID: \"3c34dfac-f28d-420f-a6da-8279cae95313\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" Nov 02 23:12:42.117426 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:42.117357 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c34dfac-f28d-420f-a6da-8279cae95313-istio-podinfo\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl\" (UID: \"3c34dfac-f28d-420f-a6da-8279cae95313\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" Nov 02 23:12:42.117507 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:42.117481 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c34dfac-f28d-420f-a6da-8279cae95313-workload-certs\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl\" (UID: \"3c34dfac-f28d-420f-a6da-8279cae95313\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" Nov 02 23:12:42.117598 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:42.117522 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c34dfac-f28d-420f-a6da-8279cae95313-credential-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl\" (UID: \"3c34dfac-f28d-420f-a6da-8279cae95313\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" Nov 02 23:12:42.117598 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:42.117587 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c34dfac-f28d-420f-a6da-8279cae95313-istio-envoy\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl\" (UID: \"3c34dfac-f28d-420f-a6da-8279cae95313\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" Nov 02 23:12:42.117709 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:42.117621 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c34dfac-f28d-420f-a6da-8279cae95313-workload-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl\" (UID: \"3c34dfac-f28d-420f-a6da-8279cae95313\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" Nov 02 23:12:42.117709 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:42.117698 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c34dfac-f28d-420f-a6da-8279cae95313-istio-data\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl\" (UID: \"3c34dfac-f28d-420f-a6da-8279cae95313\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" Nov 02 23:12:42.117814 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:42.117714 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkgsw\" (UniqueName: \"kubernetes.io/projected/3c34dfac-f28d-420f-a6da-8279cae95313-kube-api-access-mkgsw\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl\" (UID: \"3c34dfac-f28d-420f-a6da-8279cae95313\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" Nov 02 23:12:42.117877 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:42.117849 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c34dfac-f28d-420f-a6da-8279cae95313-workload-certs\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl\" (UID: \"3c34dfac-f28d-420f-a6da-8279cae95313\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" Nov 02 23:12:42.118049 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:42.118028 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c34dfac-f28d-420f-a6da-8279cae95313-workload-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl\" (UID: \"3c34dfac-f28d-420f-a6da-8279cae95313\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" Nov 02 23:12:42.118118 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:42.118079 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c34dfac-f28d-420f-a6da-8279cae95313-istiod-ca-cert\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl\" (UID: \"3c34dfac-f28d-420f-a6da-8279cae95313\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" Nov 02 23:12:42.119676 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:42.119655 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c34dfac-f28d-420f-a6da-8279cae95313-istio-envoy\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl\" (UID: \"3c34dfac-f28d-420f-a6da-8279cae95313\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" Nov 02 23:12:42.119909 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:42.119878 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c34dfac-f28d-420f-a6da-8279cae95313-istio-podinfo\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl\" (UID: \"3c34dfac-f28d-420f-a6da-8279cae95313\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" Nov 02 23:12:42.171997 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:42.171960 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c34dfac-f28d-420f-a6da-8279cae95313-istio-token\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl\" (UID: \"3c34dfac-f28d-420f-a6da-8279cae95313\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" Nov 02 23:12:42.172156 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:42.172065 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkgsw\" (UniqueName: \"kubernetes.io/projected/3c34dfac-f28d-420f-a6da-8279cae95313-kube-api-access-mkgsw\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl\" (UID: \"3c34dfac-f28d-420f-a6da-8279cae95313\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" Nov 02 23:12:42.281489 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:42.281396 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" Nov 02 23:12:42.698559 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:12:42.698530 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c34dfac_f28d_420f_a6da_8279cae95313.slice/crio-df1687a185d743959f1035cc4688a977bb0cd7fb2830c08883f46538b57cd2a5 WatchSource:0}: Error finding container df1687a185d743959f1035cc4688a977bb0cd7fb2830c08883f46538b57cd2a5: Status 404 returned error can't find the container with id df1687a185d743959f1035cc4688a977bb0cd7fb2830c08883f46538b57cd2a5 Nov 02 23:12:42.698926 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:42.698902 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl"] Nov 02 23:12:42.701008 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:42.700979 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:12:42.701076 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:42.701038 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:12:42.701076 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:42.701064 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:12:43.087993 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:43.087908 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-5m6f7-d68m6"] Nov 02 23:12:43.090987 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:43.090966 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-5m6f7-d68m6" Nov 02 23:12:43.143509 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:43.143087 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-nxb2c\"" Nov 02 23:12:43.152703 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:43.152671 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-5m6f7-d68m6"] Nov 02 23:12:43.168674 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:43.168632 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" event={"ID":"3c34dfac-f28d-420f-a6da-8279cae95313","Type":"ContainerStarted","Data":"c254545bd149977bcc76c27131e97046ed111ee0fbbd23160cccec8b213b0685"} Nov 02 23:12:43.168868 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:43.168688 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" event={"ID":"3c34dfac-f28d-420f-a6da-8279cae95313","Type":"ContainerStarted","Data":"df1687a185d743959f1035cc4688a977bb0cd7fb2830c08883f46538b57cd2a5"} Nov 02 23:12:43.239919 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:43.235137 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/06e1a697-3cf2-47d6-9a09-5177da55ac16-archives-path\") pod \"periodic-gathering-5m6f7-d68m6\" (UID: \"06e1a697-3cf2-47d6-9a09-5177da55ac16\") " pod="openshift-insights/periodic-gathering-5m6f7-d68m6" Nov 02 23:12:43.239919 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:43.235482 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhm6z\" (UniqueName: \"kubernetes.io/projected/06e1a697-3cf2-47d6-9a09-5177da55ac16-kube-api-access-vhm6z\") pod \"periodic-gathering-5m6f7-d68m6\" (UID: \"06e1a697-3cf2-47d6-9a09-5177da55ac16\") " pod="openshift-insights/periodic-gathering-5m6f7-d68m6" Nov 02 23:12:43.239919 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:43.235588 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06e1a697-3cf2-47d6-9a09-5177da55ac16-service-ca-bundle\") pod \"periodic-gathering-5m6f7-d68m6\" (UID: \"06e1a697-3cf2-47d6-9a09-5177da55ac16\") " pod="openshift-insights/periodic-gathering-5m6f7-d68m6" Nov 02 23:12:43.239919 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:43.235783 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06e1a697-3cf2-47d6-9a09-5177da55ac16-trusted-ca-bundle\") pod \"periodic-gathering-5m6f7-d68m6\" (UID: \"06e1a697-3cf2-47d6-9a09-5177da55ac16\") " pod="openshift-insights/periodic-gathering-5m6f7-d68m6" Nov 02 23:12:43.282241 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:43.282201 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" Nov 02 23:12:43.284138 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:43.284107 5076 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 02 23:12:43.284255 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:43.284154 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" podUID="3c34dfac-f28d-420f-a6da-8279cae95313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 02 23:12:43.336512 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:43.336394 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" podStartSLOduration=2.336329231 podStartE2EDuration="2.336329231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:12:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:12:43.334332379 +0000 UTC m=+1212.302864166" watchObservedRunningTime="2026-11-02 23:12:43.336329231 +0000 UTC m=+1212.304861017" Nov 02 23:12:43.337421 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:43.337391 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhm6z\" (UniqueName: \"kubernetes.io/projected/06e1a697-3cf2-47d6-9a09-5177da55ac16-kube-api-access-vhm6z\") pod \"periodic-gathering-5m6f7-d68m6\" (UID: \"06e1a697-3cf2-47d6-9a09-5177da55ac16\") " pod="openshift-insights/periodic-gathering-5m6f7-d68m6" Nov 02 23:12:43.337570 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:43.337464 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06e1a697-3cf2-47d6-9a09-5177da55ac16-service-ca-bundle\") pod \"periodic-gathering-5m6f7-d68m6\" (UID: \"06e1a697-3cf2-47d6-9a09-5177da55ac16\") " pod="openshift-insights/periodic-gathering-5m6f7-d68m6" Nov 02 23:12:43.337628 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:43.337572 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06e1a697-3cf2-47d6-9a09-5177da55ac16-trusted-ca-bundle\") pod \"periodic-gathering-5m6f7-d68m6\" (UID: \"06e1a697-3cf2-47d6-9a09-5177da55ac16\") " pod="openshift-insights/periodic-gathering-5m6f7-d68m6" Nov 02 23:12:43.337628 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:43.337599 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/06e1a697-3cf2-47d6-9a09-5177da55ac16-archives-path\") pod \"periodic-gathering-5m6f7-d68m6\" (UID: \"06e1a697-3cf2-47d6-9a09-5177da55ac16\") " pod="openshift-insights/periodic-gathering-5m6f7-d68m6" Nov 02 23:12:43.338187 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:43.338140 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/06e1a697-3cf2-47d6-9a09-5177da55ac16-archives-path\") pod \"periodic-gathering-5m6f7-d68m6\" (UID: \"06e1a697-3cf2-47d6-9a09-5177da55ac16\") " pod="openshift-insights/periodic-gathering-5m6f7-d68m6" Nov 02 23:12:43.338304 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:43.338285 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06e1a697-3cf2-47d6-9a09-5177da55ac16-service-ca-bundle\") pod \"periodic-gathering-5m6f7-d68m6\" (UID: \"06e1a697-3cf2-47d6-9a09-5177da55ac16\") " pod="openshift-insights/periodic-gathering-5m6f7-d68m6" Nov 02 23:12:43.338454 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:43.338435 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06e1a697-3cf2-47d6-9a09-5177da55ac16-trusted-ca-bundle\") pod \"periodic-gathering-5m6f7-d68m6\" (UID: \"06e1a697-3cf2-47d6-9a09-5177da55ac16\") " pod="openshift-insights/periodic-gathering-5m6f7-d68m6" Nov 02 23:12:43.392879 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:43.392849 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhm6z\" (UniqueName: \"kubernetes.io/projected/06e1a697-3cf2-47d6-9a09-5177da55ac16-kube-api-access-vhm6z\") pod \"periodic-gathering-5m6f7-d68m6\" (UID: \"06e1a697-3cf2-47d6-9a09-5177da55ac16\") " pod="openshift-insights/periodic-gathering-5m6f7-d68m6" Nov 02 23:12:43.405707 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:43.405674 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-5m6f7-d68m6" Nov 02 23:12:43.802409 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:43.802377 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-5m6f7-d68m6"] Nov 02 23:12:43.804606 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:12:43.804575 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e1a697_3cf2_47d6_9a09_5177da55ac16.slice/crio-6b2ee05f445dbd781df4cb013cab5c1e010fd76e3c01b2aa127bbed2dbbe4f23 WatchSource:0}: Error finding container 6b2ee05f445dbd781df4cb013cab5c1e010fd76e3c01b2aa127bbed2dbbe4f23: Status 404 returned error can't find the container with id 6b2ee05f445dbd781df4cb013cab5c1e010fd76e3c01b2aa127bbed2dbbe4f23 Nov 02 23:12:44.177167 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:44.177074 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-5m6f7-d68m6" event={"ID":"06e1a697-3cf2-47d6-9a09-5177da55ac16","Type":"ContainerStarted","Data":"b6e3d1a41bdc82805fd8561c00158a8e2b7d1ed134fee8e369b1291fd725fb7d"} Nov 02 23:12:44.177167 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:44.177137 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-5m6f7-d68m6" event={"ID":"06e1a697-3cf2-47d6-9a09-5177da55ac16","Type":"ContainerStarted","Data":"6b2ee05f445dbd781df4cb013cab5c1e010fd76e3c01b2aa127bbed2dbbe4f23"} Nov 02 23:12:44.282028 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:44.281983 5076 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 02 23:12:44.282203 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:44.282057 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" podUID="3c34dfac-f28d-420f-a6da-8279cae95313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 02 23:12:44.291716 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:44.291660 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-5m6f7-d68m6" podStartSLOduration=1.29163986 podStartE2EDuration="1.29163986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:12:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:12:44.289266588 +0000 UTC m=+1213.257798372" watchObservedRunningTime="2026-11-02 23:12:44.29163986 +0000 UTC m=+1213.260171643" Nov 02 23:12:45.282023 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:45.281992 5076 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 02 23:12:45.282492 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:45.282049 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" podUID="3c34dfac-f28d-420f-a6da-8279cae95313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 02 23:12:46.282760 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:46.282725 5076 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 02 23:12:46.283262 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:46.282778 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" podUID="3c34dfac-f28d-420f-a6da-8279cae95313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 02 23:12:46.735759 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:46.735721 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw"] Nov 02 23:12:46.740423 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:46.740401 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" Nov 02 23:12:46.797772 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:46.797731 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5662cc68-8f5b-48f8-8ed9-858e5595524d-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw\" (UID: \"5662cc68-8f5b-48f8-8ed9-858e5595524d\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" Nov 02 23:12:46.798106 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:46.798081 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5662cc68-8f5b-48f8-8ed9-858e5595524d-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw\" (UID: \"5662cc68-8f5b-48f8-8ed9-858e5595524d\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" Nov 02 23:12:46.798249 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:46.798232 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5662cc68-8f5b-48f8-8ed9-858e5595524d-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw\" (UID: \"5662cc68-8f5b-48f8-8ed9-858e5595524d\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" Nov 02 23:12:46.798406 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:46.798387 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5662cc68-8f5b-48f8-8ed9-858e5595524d-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw\" (UID: \"5662cc68-8f5b-48f8-8ed9-858e5595524d\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" Nov 02 23:12:46.798596 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:46.798570 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5662cc68-8f5b-48f8-8ed9-858e5595524d-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw\" (UID: \"5662cc68-8f5b-48f8-8ed9-858e5595524d\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" Nov 02 23:12:46.798836 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:46.798811 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5662cc68-8f5b-48f8-8ed9-858e5595524d-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw\" (UID: \"5662cc68-8f5b-48f8-8ed9-858e5595524d\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" Nov 02 23:12:46.799230 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:46.799199 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx8b5\" (UniqueName: \"kubernetes.io/projected/5662cc68-8f5b-48f8-8ed9-858e5595524d-kube-api-access-zx8b5\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw\" (UID: \"5662cc68-8f5b-48f8-8ed9-858e5595524d\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" Nov 02 23:12:46.799940 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:46.799614 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5662cc68-8f5b-48f8-8ed9-858e5595524d-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw\" (UID: \"5662cc68-8f5b-48f8-8ed9-858e5595524d\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" Nov 02 23:12:46.799940 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:46.799801 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5662cc68-8f5b-48f8-8ed9-858e5595524d-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw\" (UID: \"5662cc68-8f5b-48f8-8ed9-858e5595524d\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" Nov 02 23:12:46.806617 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:46.806590 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw"] Nov 02 23:12:46.901039 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:46.901005 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5662cc68-8f5b-48f8-8ed9-858e5595524d-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw\" (UID: \"5662cc68-8f5b-48f8-8ed9-858e5595524d\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" Nov 02 23:12:46.901226 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:46.901049 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5662cc68-8f5b-48f8-8ed9-858e5595524d-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw\" (UID: \"5662cc68-8f5b-48f8-8ed9-858e5595524d\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" Nov 02 23:12:46.901226 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:46.901104 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5662cc68-8f5b-48f8-8ed9-858e5595524d-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw\" (UID: \"5662cc68-8f5b-48f8-8ed9-858e5595524d\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" Nov 02 23:12:46.901226 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:46.901166 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5662cc68-8f5b-48f8-8ed9-858e5595524d-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw\" (UID: \"5662cc68-8f5b-48f8-8ed9-858e5595524d\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" Nov 02 23:12:46.901384 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:46.901265 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5662cc68-8f5b-48f8-8ed9-858e5595524d-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw\" (UID: \"5662cc68-8f5b-48f8-8ed9-858e5595524d\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" Nov 02 23:12:46.901384 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:46.901303 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx8b5\" (UniqueName: \"kubernetes.io/projected/5662cc68-8f5b-48f8-8ed9-858e5595524d-kube-api-access-zx8b5\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw\" (UID: \"5662cc68-8f5b-48f8-8ed9-858e5595524d\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" Nov 02 23:12:46.901384 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:46.901345 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5662cc68-8f5b-48f8-8ed9-858e5595524d-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw\" (UID: \"5662cc68-8f5b-48f8-8ed9-858e5595524d\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" Nov 02 23:12:46.901534 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:46.901428 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5662cc68-8f5b-48f8-8ed9-858e5595524d-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw\" (UID: \"5662cc68-8f5b-48f8-8ed9-858e5595524d\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" Nov 02 23:12:46.901534 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:46.901492 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5662cc68-8f5b-48f8-8ed9-858e5595524d-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw\" (UID: \"5662cc68-8f5b-48f8-8ed9-858e5595524d\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" Nov 02 23:12:46.901680 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:46.901546 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5662cc68-8f5b-48f8-8ed9-858e5595524d-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw\" (UID: \"5662cc68-8f5b-48f8-8ed9-858e5595524d\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" Nov 02 23:12:46.902059 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:46.902035 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5662cc68-8f5b-48f8-8ed9-858e5595524d-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw\" (UID: \"5662cc68-8f5b-48f8-8ed9-858e5595524d\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" Nov 02 23:12:46.902172 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:46.902076 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5662cc68-8f5b-48f8-8ed9-858e5595524d-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw\" (UID: \"5662cc68-8f5b-48f8-8ed9-858e5595524d\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" Nov 02 23:12:46.902300 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:46.902277 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5662cc68-8f5b-48f8-8ed9-858e5595524d-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw\" (UID: \"5662cc68-8f5b-48f8-8ed9-858e5595524d\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" Nov 02 23:12:46.902554 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:46.902530 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5662cc68-8f5b-48f8-8ed9-858e5595524d-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw\" (UID: \"5662cc68-8f5b-48f8-8ed9-858e5595524d\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" Nov 02 23:12:46.903706 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:46.903681 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5662cc68-8f5b-48f8-8ed9-858e5595524d-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw\" (UID: \"5662cc68-8f5b-48f8-8ed9-858e5595524d\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" Nov 02 23:12:46.904532 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:46.904508 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5662cc68-8f5b-48f8-8ed9-858e5595524d-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw\" (UID: \"5662cc68-8f5b-48f8-8ed9-858e5595524d\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" Nov 02 23:12:46.956591 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:46.955745 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx8b5\" (UniqueName: \"kubernetes.io/projected/5662cc68-8f5b-48f8-8ed9-858e5595524d-kube-api-access-zx8b5\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw\" (UID: \"5662cc68-8f5b-48f8-8ed9-858e5595524d\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" Nov 02 23:12:46.956591 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:46.955946 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5662cc68-8f5b-48f8-8ed9-858e5595524d-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw\" (UID: \"5662cc68-8f5b-48f8-8ed9-858e5595524d\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" Nov 02 23:12:47.058614 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:47.058532 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" Nov 02 23:12:47.282077 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:47.282042 5076 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 02 23:12:47.282252 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:47.282094 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" podUID="3c34dfac-f28d-420f-a6da-8279cae95313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 02 23:12:47.537609 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:47.537563 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw"] Nov 02 23:12:47.541229 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:12:47.541200 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5662cc68_8f5b_48f8_8ed9_858e5595524d.slice/crio-bbddb1980d2d15a3a360886c5b4f529c0b1cfa77683747955f4e782f24f9d585 WatchSource:0}: Error finding container bbddb1980d2d15a3a360886c5b4f529c0b1cfa77683747955f4e782f24f9d585: Status 404 returned error can't find the container with id bbddb1980d2d15a3a360886c5b4f529c0b1cfa77683747955f4e782f24f9d585 Nov 02 23:12:47.543531 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:47.543500 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:12:47.543685 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:47.543670 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:12:47.543785 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:47.543773 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:12:48.206930 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:48.206883 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" event={"ID":"5662cc68-8f5b-48f8-8ed9-858e5595524d","Type":"ContainerStarted","Data":"240f24a25f4ee2b62df78be0ac271323c6968f322aba72c0f11082bd116980bc"} Nov 02 23:12:48.207121 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:48.206955 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" event={"ID":"5662cc68-8f5b-48f8-8ed9-858e5595524d","Type":"ContainerStarted","Data":"bbddb1980d2d15a3a360886c5b4f529c0b1cfa77683747955f4e782f24f9d585"} Nov 02 23:12:48.282108 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:48.282060 5076 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 02 23:12:48.282306 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:48.282171 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" podUID="3c34dfac-f28d-420f-a6da-8279cae95313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 02 23:12:48.384675 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:48.384616 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" podStartSLOduration=2.38459973 podStartE2EDuration="2.38459973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:12:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:12:48.365366862 +0000 UTC m=+1217.333898652" watchObservedRunningTime="2026-11-02 23:12:48.38459973 +0000 UTC m=+1217.353131514" Nov 02 23:12:49.059653 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:49.059610 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" Nov 02 23:12:49.061381 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:49.061350 5076 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 02 23:12:49.061536 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:49.061424 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" podUID="5662cc68-8f5b-48f8-8ed9-858e5595524d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 02 23:12:49.304297 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:49.304264 5076 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 02 23:12:49.304609 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:49.304546 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" podUID="3c34dfac-f28d-420f-a6da-8279cae95313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 02 23:12:50.059138 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:50.059108 5076 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 02 23:12:50.059328 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:50.059160 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" podUID="5662cc68-8f5b-48f8-8ed9-858e5595524d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 02 23:12:50.346307 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:50.346220 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" Nov 02 23:12:50.347198 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:50.347171 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" Nov 02 23:12:50.348068 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:50.348048 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" Nov 02 23:12:51.059817 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:51.059783 5076 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 02 23:12:51.060038 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:51.059838 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" podUID="5662cc68-8f5b-48f8-8ed9-858e5595524d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 02 23:12:52.062378 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:52.062336 5076 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 02 23:12:52.063018 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:52.062984 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" podUID="5662cc68-8f5b-48f8-8ed9-858e5595524d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 02 23:12:53.064913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:53.064363 5076 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 02 23:12:53.064913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:53.064428 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" podUID="5662cc68-8f5b-48f8-8ed9-858e5595524d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 02 23:12:54.071922 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:54.071480 5076 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 02 23:12:54.071922 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:54.071539 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" podUID="5662cc68-8f5b-48f8-8ed9-858e5595524d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 02 23:12:54.660969 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:54.633086 5076 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:12:54.660969 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:54.633158 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" podUID="0450c8f1-c6f6-4d65-a0a0-7e954133793d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:55.070846 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:55.070405 5076 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 02 23:12:55.070846 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:55.070469 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" podUID="5662cc68-8f5b-48f8-8ed9-858e5595524d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 02 23:12:55.465724 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:55.465665 5076 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:12:55.467603 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:55.467555 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" podUID="dc8027a0-9041-42ec-af51-46526e00811f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:56.066176 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:56.066140 5076 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 02 23:12:56.090121 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:56.090066 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" podUID="5662cc68-8f5b-48f8-8ed9-858e5595524d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 02 23:12:56.196735 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:56.196691 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" podUID="cf2b01c6-8001-49e3-908a-38cffb88edfa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Nov 02 23:12:57.079934 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:57.075318 5076 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 02 23:12:57.079934 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:57.075396 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" podUID="5662cc68-8f5b-48f8-8ed9-858e5595524d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 02 23:12:57.841973 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:57.839016 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" podUID="c5cd8831-584f-47bb-9580-4d41deab29dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:58.063043 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:58.063005 5076 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 02 23:12:58.083926 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:58.063295 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" podUID="5662cc68-8f5b-48f8-8ed9-858e5595524d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 02 23:12:59.059216 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:59.059184 5076 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 02 23:12:59.059379 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:12:59.059238 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" podUID="5662cc68-8f5b-48f8-8ed9-858e5595524d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 02 23:13:00.623025 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:00.622992 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" Nov 02 23:13:00.624250 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:00.624222 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" Nov 02 23:13:00.625493 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:00.625460 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" Nov 02 23:13:09.883748 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:13:09.883711 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:13:12.227358 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:12.227326 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz"] Nov 02 23:13:12.233340 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:12.233309 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" Nov 02 23:13:12.334950 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:12.334874 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz"] Nov 02 23:13:12.379963 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:12.379923 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-istio-podinfo\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz\" (UID: \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" Nov 02 23:13:12.380152 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:12.379974 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-istio-token\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz\" (UID: \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" Nov 02 23:13:12.380152 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:12.380031 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-istiod-ca-cert\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz\" (UID: \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" Nov 02 23:13:12.380620 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:12.380593 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-credential-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz\" (UID: \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" Nov 02 23:13:12.380707 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:12.380659 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-workload-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz\" (UID: \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" Nov 02 23:13:12.380768 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:12.380726 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-workload-certs\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz\" (UID: \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" Nov 02 23:13:12.380825 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:12.380807 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-istio-envoy\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz\" (UID: \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" Nov 02 23:13:12.380905 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:12.380833 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-istio-data\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz\" (UID: \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" Nov 02 23:13:12.380905 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:12.380873 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x9t8\" (UniqueName: \"kubernetes.io/projected/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-kube-api-access-7x9t8\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz\" (UID: \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" Nov 02 23:13:12.482029 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:12.481945 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-istiod-ca-cert\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz\" (UID: \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" Nov 02 23:13:12.482174 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:12.482055 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-credential-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz\" (UID: \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" Nov 02 23:13:12.482174 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:12.482109 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-workload-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz\" (UID: \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" Nov 02 23:13:12.482267 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:12.482170 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-workload-certs\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz\" (UID: \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" Nov 02 23:13:12.482267 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:12.482256 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-istio-envoy\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz\" (UID: \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" Nov 02 23:13:12.482366 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:12.482281 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-istio-data\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz\" (UID: \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" Nov 02 23:13:12.482366 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:12.482322 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x9t8\" (UniqueName: \"kubernetes.io/projected/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-kube-api-access-7x9t8\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz\" (UID: \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" Nov 02 23:13:12.482438 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:12.482427 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-istio-podinfo\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz\" (UID: \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" Nov 02 23:13:12.482479 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:12.482452 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-istio-token\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz\" (UID: \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" Nov 02 23:13:12.483020 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:12.482995 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-credential-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz\" (UID: \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" Nov 02 23:13:12.483114 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:12.483032 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-istiod-ca-cert\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz\" (UID: \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" Nov 02 23:13:12.483114 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:12.483071 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-workload-certs\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz\" (UID: \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" Nov 02 23:13:12.483114 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:12.483037 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-workload-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz\" (UID: \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" Nov 02 23:13:12.483267 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:12.483111 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-istio-data\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz\" (UID: \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" Nov 02 23:13:12.485098 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:12.485072 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-istio-envoy\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz\" (UID: \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" Nov 02 23:13:12.485380 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:12.485359 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-istio-podinfo\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz\" (UID: \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" Nov 02 23:13:12.563442 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:12.563366 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x9t8\" (UniqueName: \"kubernetes.io/projected/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-kube-api-access-7x9t8\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz\" (UID: \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" Nov 02 23:13:12.564471 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:12.564444 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-istio-token\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz\" (UID: \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" Nov 02 23:13:12.858671 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:12.858563 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" Nov 02 23:13:13.305815 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:13:13.305779 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f9a57c0_49f1_435b_8c3b_d64bfe5a1efe.slice/crio-591d35ab3f12a96e87235d6ba6d85528300cea12756e5c7cb1d7238c3cd1620a WatchSource:0}: Error finding container 591d35ab3f12a96e87235d6ba6d85528300cea12756e5c7cb1d7238c3cd1620a: Status 404 returned error can't find the container with id 591d35ab3f12a96e87235d6ba6d85528300cea12756e5c7cb1d7238c3cd1620a Nov 02 23:13:13.306231 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:13.305994 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz"] Nov 02 23:13:13.308219 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:13.308186 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:13:13.308312 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:13.308252 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:13:13.308312 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:13.308278 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:13:13.690205 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:13.690170 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" event={"ID":"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe","Type":"ContainerStarted","Data":"8a09c2709748659f0f63de0be865e5fb0e44a25a3aeef856e214c5e6b569e360"} Nov 02 23:13:13.690383 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:13.690234 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" event={"ID":"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe","Type":"ContainerStarted","Data":"591d35ab3f12a96e87235d6ba6d85528300cea12756e5c7cb1d7238c3cd1620a"} Nov 02 23:13:13.803669 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:13.803596 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" podStartSLOduration=1.803581287 podStartE2EDuration="1.803581287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:13:13.801289401 +0000 UTC m=+1242.769821185" watchObservedRunningTime="2026-11-02 23:13:13.803581287 +0000 UTC m=+1242.772113072" Nov 02 23:13:13.865921 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:13.862988 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" Nov 02 23:13:14.861957 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:14.860737 5076 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 02 23:13:14.861957 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:14.860829 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" podUID="7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 02 23:13:15.859067 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:15.859022 5076 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 02 23:13:15.859282 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:15.859135 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" podUID="7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 02 23:13:16.859360 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:16.859325 5076 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 02 23:13:16.859806 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:16.859380 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" podUID="7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 02 23:13:17.859371 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:17.859338 5076 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 02 23:13:17.859790 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:17.859399 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" podUID="7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 02 23:13:18.449271 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:18.449229 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vl8q5"] Nov 02 23:13:18.474256 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:18.474223 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vl8q5" Nov 02 23:13:18.502794 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:18.494208 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vl8q5"] Nov 02 23:13:18.504374 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:18.504037 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c00852d-1d5f-4caf-9df9-78c76b570a77-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e953257b-6756fc6855-vl8q5\" (UID: \"6c00852d-1d5f-4caf-9df9-78c76b570a77\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vl8q5" Nov 02 23:13:18.504374 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:18.504163 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86wn7\" (UniqueName: \"kubernetes.io/projected/6c00852d-1d5f-4caf-9df9-78c76b570a77-kube-api-access-86wn7\") pod \"payload-pre-processing-e2e-mt-e953257b-6756fc6855-vl8q5\" (UID: \"6c00852d-1d5f-4caf-9df9-78c76b570a77\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vl8q5" Nov 02 23:13:18.560686 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:18.559349 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e953257b\"" Nov 02 23:13:18.612257 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:18.612221 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e953257b-dockercfg-bmwt4\"" Nov 02 23:13:18.639120 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:18.639065 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c00852d-1d5f-4caf-9df9-78c76b570a77-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e953257b-6756fc6855-vl8q5\" (UID: \"6c00852d-1d5f-4caf-9df9-78c76b570a77\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vl8q5" Nov 02 23:13:18.670969 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:18.647040 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86wn7\" (UniqueName: \"kubernetes.io/projected/6c00852d-1d5f-4caf-9df9-78c76b570a77-kube-api-access-86wn7\") pod \"payload-pre-processing-e2e-mt-e953257b-6756fc6855-vl8q5\" (UID: \"6c00852d-1d5f-4caf-9df9-78c76b570a77\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vl8q5" Nov 02 23:13:18.713993 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:18.711936 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c00852d-1d5f-4caf-9df9-78c76b570a77-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e953257b-6756fc6855-vl8q5\" (UID: \"6c00852d-1d5f-4caf-9df9-78c76b570a77\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vl8q5" Nov 02 23:13:18.885713 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:18.885583 5076 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 02 23:13:18.886267 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:18.886232 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" podUID="7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 02 23:13:18.886820 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:18.886794 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86wn7\" (UniqueName: \"kubernetes.io/projected/6c00852d-1d5f-4caf-9df9-78c76b570a77-kube-api-access-86wn7\") pod \"payload-pre-processing-e2e-mt-e953257b-6756fc6855-vl8q5\" (UID: \"6c00852d-1d5f-4caf-9df9-78c76b570a77\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vl8q5" Nov 02 23:13:18.888351 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:18.888331 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vl8q5" Nov 02 23:13:19.404517 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:19.398536 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vl8q5"] Nov 02 23:13:19.860361 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:19.859826 5076 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 02 23:13:19.860361 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:19.859943 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" podUID="7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 02 23:13:19.961933 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:19.961877 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vl8q5" event={"ID":"6c00852d-1d5f-4caf-9df9-78c76b570a77","Type":"ContainerStarted","Data":"b869936754d158cae930bedc183c9d38d53e500e13ad5d37070f4f999a789ea6"} Nov 02 23:13:20.704048 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:20.703992 5076 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" podUID="c5cd8831-584f-47bb-9580-4d41deab29dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": context deadline exceeded" Nov 02 23:13:20.925035 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:20.924997 5076 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 02 23:13:20.925328 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:20.925297 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" podUID="7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 02 23:13:21.055990 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:21.055807 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vl8q5" event={"ID":"6c00852d-1d5f-4caf-9df9-78c76b570a77","Type":"ContainerStarted","Data":"dcbc448dbabf8800a65ceee3c9783f45ee21a2b610f769f85f787275f444d8dc"} Nov 02 23:13:21.057860 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:21.057836 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vl8q5" Nov 02 23:13:21.896915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:21.893272 5076 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 02 23:13:21.896915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:21.893335 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" podUID="7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 02 23:13:22.297092 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:22.297007 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" podUID="cf2b01c6-8001-49e3-908a-38cffb88edfa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:22.884348 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:22.884313 5076 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 02 23:13:22.884589 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:22.884562 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" podUID="7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 02 23:13:23.465970 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:23.464073 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" podUID="cf2b01c6-8001-49e3-908a-38cffb88edfa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Nov 02 23:13:23.481407 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:23.481374 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" Nov 02 23:13:23.719973 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:23.712090 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vl8q5" podStartSLOduration=5.712068214 podStartE2EDuration="5.712068214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:13:21.20462705 +0000 UTC m=+1250.173158835" watchObservedRunningTime="2026-11-02 23:13:23.712068214 +0000 UTC m=+1252.680599999" Nov 02 23:13:23.862207 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:23.862164 5076 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 02 23:13:23.862514 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:23.862469 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" podUID="7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 02 23:13:23.872836 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:23.872782 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" podUID="c5cd8831-584f-47bb-9580-4d41deab29dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:23.947943 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:23.947865 5076 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:13:23.948295 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:23.948259 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" podUID="2cfe1803-0eec-4193-aa8f-05b3699a0ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:24.329915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:24.328988 5076 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:13:24.329915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:24.329051 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" podUID="ca62790c-9f83-45df-9bb8-6360b271dab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:24.471355 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:24.471310 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" podUID="cf2b01c6-8001-49e3-908a-38cffb88edfa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:24.535083 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:24.534919 5076 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:13:24.535083 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:24.534990 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" podUID="0450c8f1-c6f6-4d65-a0a0-7e954133793d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:24.884917 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:24.866021 5076 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 02 23:13:24.884917 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:24.866084 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" podUID="7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 02 23:13:25.036917 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:25.012252 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" podUID="c5cd8831-584f-47bb-9580-4d41deab29dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:25.087915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:25.051303 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" Nov 02 23:13:25.856916 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:25.828042 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" podUID="cf2b01c6-8001-49e3-908a-38cffb88edfa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:26.056920 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:26.039354 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" podUID="c5cd8831-584f-47bb-9580-4d41deab29dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:26.344969 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:26.337409 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:13:26.344969 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:26.337475 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" podUID="a67b3447-b8b2-4485-bacf-0d39a1352feb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:26.504915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:26.490381 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" Nov 02 23:13:26.647466 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:26.647056 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:13:26.647466 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:26.647120 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" podUID="db841b5b-fade-48e4-9515-ef923b888f42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:26.711761 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:26.711727 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" Nov 02 23:13:26.903227 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:26.903110 5076 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:13:26.903825 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:26.903775 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" podUID="7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:27.859919 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:27.859442 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" Nov 02 23:13:27.947061 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:27.947018 5076 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:13:27.947463 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:27.947086 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" podUID="7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:28.493775 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:28.493738 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" Nov 02 23:13:28.495394 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:28.495367 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" Nov 02 23:13:33.632351 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:33.632321 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vl8q5" Nov 02 23:13:39.887138 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:13:39.887097 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:13:43.982625 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:43.982586 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl"] Nov 02 23:13:43.988268 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:43.988248 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl" Nov 02 23:13:44.040372 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:44.040329 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl"] Nov 02 23:13:44.083255 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:44.083222 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e953257b-serving-cert\"" Nov 02 23:13:44.119339 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:44.119306 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/675b1902-02f6-49c3-bd75-1feecd543608-metrics-certs\") pod \"maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl\" (UID: \"675b1902-02f6-49c3-bd75-1feecd543608\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl" Nov 02 23:13:44.119513 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:44.119356 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/675b1902-02f6-49c3-bd75-1feecd543608-openshift-service-ca\") pod \"maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl\" (UID: \"675b1902-02f6-49c3-bd75-1feecd543608\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl" Nov 02 23:13:44.119599 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:44.119573 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlr5h\" (UniqueName: \"kubernetes.io/projected/675b1902-02f6-49c3-bd75-1feecd543608-kube-api-access-vlr5h\") pod \"maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl\" (UID: \"675b1902-02f6-49c3-bd75-1feecd543608\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl" Nov 02 23:13:44.119668 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:44.119630 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/675b1902-02f6-49c3-bd75-1feecd543608-maas-api-tls\") pod \"maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl\" (UID: \"675b1902-02f6-49c3-bd75-1feecd543608\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl" Nov 02 23:13:44.220958 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:44.220878 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/675b1902-02f6-49c3-bd75-1feecd543608-metrics-certs\") pod \"maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl\" (UID: \"675b1902-02f6-49c3-bd75-1feecd543608\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl" Nov 02 23:13:44.221142 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:44.220967 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/675b1902-02f6-49c3-bd75-1feecd543608-openshift-service-ca\") pod \"maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl\" (UID: \"675b1902-02f6-49c3-bd75-1feecd543608\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl" Nov 02 23:13:44.221142 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:44.221093 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlr5h\" (UniqueName: \"kubernetes.io/projected/675b1902-02f6-49c3-bd75-1feecd543608-kube-api-access-vlr5h\") pod \"maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl\" (UID: \"675b1902-02f6-49c3-bd75-1feecd543608\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl" Nov 02 23:13:44.221142 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:44.221126 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/675b1902-02f6-49c3-bd75-1feecd543608-maas-api-tls\") pod \"maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl\" (UID: \"675b1902-02f6-49c3-bd75-1feecd543608\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl" Nov 02 23:13:44.221718 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:44.221689 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/675b1902-02f6-49c3-bd75-1feecd543608-openshift-service-ca\") pod \"maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl\" (UID: \"675b1902-02f6-49c3-bd75-1feecd543608\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl" Nov 02 23:13:44.223483 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:44.223459 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/675b1902-02f6-49c3-bd75-1feecd543608-metrics-certs\") pod \"maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl\" (UID: \"675b1902-02f6-49c3-bd75-1feecd543608\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl" Nov 02 23:13:44.223590 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:44.223529 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/675b1902-02f6-49c3-bd75-1feecd543608-maas-api-tls\") pod \"maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl\" (UID: \"675b1902-02f6-49c3-bd75-1feecd543608\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl" Nov 02 23:13:44.276606 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:44.276531 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlr5h\" (UniqueName: \"kubernetes.io/projected/675b1902-02f6-49c3-bd75-1feecd543608-kube-api-access-vlr5h\") pod \"maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl\" (UID: \"675b1902-02f6-49c3-bd75-1feecd543608\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl" Nov 02 23:13:44.302414 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:44.302375 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl" Nov 02 23:13:44.707286 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:44.707252 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl"] Nov 02 23:13:44.707683 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:13:44.707654 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod675b1902_02f6_49c3_bd75_1feecd543608.slice/crio-24314cf8ece5d91bec9460eba7270338daec057d3d8e46d2ac52d22eeba837c5 WatchSource:0}: Error finding container 24314cf8ece5d91bec9460eba7270338daec057d3d8e46d2ac52d22eeba837c5: Status 404 returned error can't find the container with id 24314cf8ece5d91bec9460eba7270338daec057d3d8e46d2ac52d22eeba837c5 Nov 02 23:13:44.709068 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:44.709052 5076 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 23:13:45.416982 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:45.416944 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl" event={"ID":"675b1902-02f6-49c3-bd75-1feecd543608","Type":"ContainerStarted","Data":"24314cf8ece5d91bec9460eba7270338daec057d3d8e46d2ac52d22eeba837c5"} Nov 02 23:13:46.429485 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:46.429443 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl" event={"ID":"675b1902-02f6-49c3-bd75-1feecd543608","Type":"ContainerStarted","Data":"fc2785c1b757ed8b71f3d8e4c439c87859dad6d8de9a966772ae57a8be4d61e3"} Nov 02 23:13:46.430062 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:46.429645 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl" Nov 02 23:13:48.843036 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:48.842973 5076 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" podUID="cf2b01c6-8001-49e3-908a-38cffb88edfa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": context deadline exceeded" Nov 02 23:13:48.889021 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:48.888970 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" podUID="cf2b01c6-8001-49e3-908a-38cffb88edfa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:50.186919 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:50.186468 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" podUID="c5cd8831-584f-47bb-9580-4d41deab29dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:50.343867 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:50.343485 5076 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:13:50.343867 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:50.343564 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" podUID="8dd885dc-9e61-4303-a916-f422ab5eb322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:50.724076 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:50.724036 5076 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" podUID="c5cd8831-584f-47bb-9580-4d41deab29dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": context deadline exceeded" Nov 02 23:13:51.454177 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:51.454115 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" podUID="c5cd8831-584f-47bb-9580-4d41deab29dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:52.464910 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:52.462637 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:13:52.464910 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:52.462719 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" podUID="ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:52.464910 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:52.464048 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" Nov 02 23:13:52.486055 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:52.486008 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" podUID="c5cd8831-584f-47bb-9580-4d41deab29dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 02 23:13:52.936999 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:52.882854 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl" podStartSLOduration=8.772130005 podStartE2EDuration="9.882834293s" totalImagesPullingTime="1.110704288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:13:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:13:46.549197133 +0000 UTC m=+1275.517728937" watchObservedRunningTime="2026-11-02 23:13:52.882834293 +0000 UTC m=+1281.851366078" Nov 02 23:13:53.073569 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:53.073537 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl" Nov 02 23:13:53.075464 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:53.075440 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" Nov 02 23:13:53.696982 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:53.696469 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-sc89z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:13:53.696982 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:13:53.696549 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" podUID="9b820762-bd67-4d19-a050-224c6225fe67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:14:05.562732 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:05.562691 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-88d4f7ff9-xsrng"] Nov 02 23:14:09.885622 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:14:09.885581 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:14:10.563826 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:10.563754 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-88d4f7ff9-xsrng" podUID="eac149ce-3ee3-4232-8b82-10eb431d939d" containerName="payload-processing" containerID="cri-o://7151f5dc86ba539512a5b553a4d76ec2d3db32a403eed92beb678d25f1404af0" gracePeriod=25 Nov 02 23:14:10.869674 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:10.869648 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-88d4f7ff9-xsrng" Nov 02 23:14:10.969622 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:10.969586 5076 generic.go:356] "Generic (PLEG): container finished" podID="eac149ce-3ee3-4232-8b82-10eb431d939d" containerID="7151f5dc86ba539512a5b553a4d76ec2d3db32a403eed92beb678d25f1404af0" exitCode=0 Nov 02 23:14:10.969622 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:10.969628 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-88d4f7ff9-xsrng" Nov 02 23:14:10.970198 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:10.969668 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-88d4f7ff9-xsrng" event={"ID":"eac149ce-3ee3-4232-8b82-10eb431d939d","Type":"ContainerDied","Data":"7151f5dc86ba539512a5b553a4d76ec2d3db32a403eed92beb678d25f1404af0"} Nov 02 23:14:10.970198 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:10.969754 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-88d4f7ff9-xsrng" event={"ID":"eac149ce-3ee3-4232-8b82-10eb431d939d","Type":"ContainerDied","Data":"f53195dbb2988b2d78768cdc86649b963576dedb15f4121464d76927d1f013a0"} Nov 02 23:14:10.970198 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:10.969794 5076 scope.go:122] "RemoveContainer" containerID="7151f5dc86ba539512a5b553a4d76ec2d3db32a403eed92beb678d25f1404af0" Nov 02 23:14:10.982276 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:10.982250 5076 scope.go:122] "RemoveContainer" containerID="7151f5dc86ba539512a5b553a4d76ec2d3db32a403eed92beb678d25f1404af0" Nov 02 23:14:10.982628 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:14:10.982606 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7151f5dc86ba539512a5b553a4d76ec2d3db32a403eed92beb678d25f1404af0\": container with ID starting with 7151f5dc86ba539512a5b553a4d76ec2d3db32a403eed92beb678d25f1404af0 not found: ID does not exist" containerID="7151f5dc86ba539512a5b553a4d76ec2d3db32a403eed92beb678d25f1404af0" Nov 02 23:14:10.982740 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:10.982638 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7151f5dc86ba539512a5b553a4d76ec2d3db32a403eed92beb678d25f1404af0"} err="failed to get container status \"7151f5dc86ba539512a5b553a4d76ec2d3db32a403eed92beb678d25f1404af0\": rpc error: code = NotFound desc = could not find container \"7151f5dc86ba539512a5b553a4d76ec2d3db32a403eed92beb678d25f1404af0\": container with ID starting with 7151f5dc86ba539512a5b553a4d76ec2d3db32a403eed92beb678d25f1404af0 not found: ID does not exist" Nov 02 23:14:10.992198 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:10.992161 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eac149ce-3ee3-4232-8b82-10eb431d939d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eac149ce-3ee3-4232-8b82-10eb431d939d-plugins-config-volume\") pod \"eac149ce-3ee3-4232-8b82-10eb431d939d\" (UID: \"eac149ce-3ee3-4232-8b82-10eb431d939d\") " Nov 02 23:14:10.992312 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:10.992230 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eac149ce-3ee3-4232-8b82-10eb431d939d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eac149ce-3ee3-4232-8b82-10eb431d939d-routing-overlay-volume\") pod \"eac149ce-3ee3-4232-8b82-10eb431d939d\" (UID: \"eac149ce-3ee3-4232-8b82-10eb431d939d\") " Nov 02 23:14:10.992312 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:10.992287 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eac149ce-3ee3-4232-8b82-10eb431d939d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/eac149ce-3ee3-4232-8b82-10eb431d939d-provider-credentials\") pod \"eac149ce-3ee3-4232-8b82-10eb431d939d\" (UID: \"eac149ce-3ee3-4232-8b82-10eb431d939d\") " Nov 02 23:14:10.992541 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:10.992523 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eac149ce-3ee3-4232-8b82-10eb431d939d-kube-api-access-kjzlr\" (UniqueName: \"kubernetes.io/projected/eac149ce-3ee3-4232-8b82-10eb431d939d-kube-api-access-kjzlr\") pod \"eac149ce-3ee3-4232-8b82-10eb431d939d\" (UID: \"eac149ce-3ee3-4232-8b82-10eb431d939d\") " Nov 02 23:14:10.992607 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:10.992583 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eac149ce-3ee3-4232-8b82-10eb431d939d-routing-overlay-volume" pod "eac149ce-3ee3-4232-8b82-10eb431d939d" (UID: "eac149ce-3ee3-4232-8b82-10eb431d939d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:14:10.992607 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:10.992597 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eac149ce-3ee3-4232-8b82-10eb431d939d-plugins-config-volume" pod "eac149ce-3ee3-4232-8b82-10eb431d939d" (UID: "eac149ce-3ee3-4232-8b82-10eb431d939d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:14:10.993160 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:10.993135 5076 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eac149ce-3ee3-4232-8b82-10eb431d939d-plugins-config-volume\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:14:10.993257 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:10.993164 5076 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eac149ce-3ee3-4232-8b82-10eb431d939d-routing-overlay-volume\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:14:10.994434 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:10.994412 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eac149ce-3ee3-4232-8b82-10eb431d939d-kube-api-access-kjzlr" pod "eac149ce-3ee3-4232-8b82-10eb431d939d" (UID: "eac149ce-3ee3-4232-8b82-10eb431d939d"). InnerVolumeSpecName "kube-api-access-kjzlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:14:10.994505 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:10.994421 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eac149ce-3ee3-4232-8b82-10eb431d939d-provider-credentials" pod "eac149ce-3ee3-4232-8b82-10eb431d939d" (UID: "eac149ce-3ee3-4232-8b82-10eb431d939d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:14:11.094328 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:11.094221 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjzlr\" (UniqueName: \"kubernetes.io/projected/eac149ce-3ee3-4232-8b82-10eb431d939d-kube-api-access-kjzlr\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:14:11.094328 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:11.094252 5076 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eac149ce-3ee3-4232-8b82-10eb431d939d-provider-credentials\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:14:11.379848 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:11.379760 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-88d4f7ff9-xsrng"] Nov 02 23:14:11.382732 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:11.382709 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-88d4f7ff9-xsrng"] Nov 02 23:14:13.149167 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:13.149135 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eac149ce-3ee3-4232-8b82-10eb431d939d" path="/var/lib/kubelet/pods/eac149ce-3ee3-4232-8b82-10eb431d939d/volumes" Nov 02 23:14:21.105818 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:21.105067 5076 generic.go:356] "Generic (PLEG): container finished" podID="06e1a697-3cf2-47d6-9a09-5177da55ac16" containerID="b6e3d1a41bdc82805fd8561c00158a8e2b7d1ed134fee8e369b1291fd725fb7d" exitCode=0 Nov 02 23:14:21.105818 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:21.105281 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-5m6f7-d68m6" event={"ID":"06e1a697-3cf2-47d6-9a09-5177da55ac16","Type":"ContainerDied","Data":"b6e3d1a41bdc82805fd8561c00158a8e2b7d1ed134fee8e369b1291fd725fb7d"} Nov 02 23:14:23.001263 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:23.001231 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-5m6f7-d68m6" Nov 02 23:14:23.045652 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:23.045611 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06e1a697-3cf2-47d6-9a09-5177da55ac16-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06e1a697-3cf2-47d6-9a09-5177da55ac16-trusted-ca-bundle\") pod \"06e1a697-3cf2-47d6-9a09-5177da55ac16\" (UID: \"06e1a697-3cf2-47d6-9a09-5177da55ac16\") " Nov 02 23:14:23.046512 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:23.046465 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06e1a697-3cf2-47d6-9a09-5177da55ac16-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/06e1a697-3cf2-47d6-9a09-5177da55ac16-archives-path\") pod \"06e1a697-3cf2-47d6-9a09-5177da55ac16\" (UID: \"06e1a697-3cf2-47d6-9a09-5177da55ac16\") " Nov 02 23:14:23.047273 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:23.047249 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06e1a697-3cf2-47d6-9a09-5177da55ac16-kube-api-access-vhm6z\" (UniqueName: \"kubernetes.io/projected/06e1a697-3cf2-47d6-9a09-5177da55ac16-kube-api-access-vhm6z\") pod \"06e1a697-3cf2-47d6-9a09-5177da55ac16\" (UID: \"06e1a697-3cf2-47d6-9a09-5177da55ac16\") " Nov 02 23:14:23.047760 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:23.047739 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06e1a697-3cf2-47d6-9a09-5177da55ac16-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06e1a697-3cf2-47d6-9a09-5177da55ac16-service-ca-bundle\") pod \"06e1a697-3cf2-47d6-9a09-5177da55ac16\" (UID: \"06e1a697-3cf2-47d6-9a09-5177da55ac16\") " Nov 02 23:14:23.048826 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:23.046361 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06e1a697-3cf2-47d6-9a09-5177da55ac16-trusted-ca-bundle" pod "06e1a697-3cf2-47d6-9a09-5177da55ac16" (UID: "06e1a697-3cf2-47d6-9a09-5177da55ac16"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:14:23.049654 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:23.047014 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e1a697-3cf2-47d6-9a09-5177da55ac16-archives-path" pod "06e1a697-3cf2-47d6-9a09-5177da55ac16" (UID: "06e1a697-3cf2-47d6-9a09-5177da55ac16"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:14:23.049787 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:23.048215 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06e1a697-3cf2-47d6-9a09-5177da55ac16-service-ca-bundle" pod "06e1a697-3cf2-47d6-9a09-5177da55ac16" (UID: "06e1a697-3cf2-47d6-9a09-5177da55ac16"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:14:23.053564 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:23.053541 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06e1a697-3cf2-47d6-9a09-5177da55ac16-kube-api-access-vhm6z" pod "06e1a697-3cf2-47d6-9a09-5177da55ac16" (UID: "06e1a697-3cf2-47d6-9a09-5177da55ac16"). InnerVolumeSpecName "kube-api-access-vhm6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:14:23.152207 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:23.152116 5076 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06e1a697-3cf2-47d6-9a09-5177da55ac16-trusted-ca-bundle\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:14:23.152431 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:23.152412 5076 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/06e1a697-3cf2-47d6-9a09-5177da55ac16-archives-path\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:14:23.152535 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:23.152522 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhm6z\" (UniqueName: \"kubernetes.io/projected/06e1a697-3cf2-47d6-9a09-5177da55ac16-kube-api-access-vhm6z\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:14:23.152629 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:23.152617 5076 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06e1a697-3cf2-47d6-9a09-5177da55ac16-service-ca-bundle\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:14:23.184248 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:23.184220 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-5m6f7-d68m6" Nov 02 23:14:23.185291 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:23.184301 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-5m6f7-d68m6" event={"ID":"06e1a697-3cf2-47d6-9a09-5177da55ac16","Type":"ContainerDied","Data":"6b2ee05f445dbd781df4cb013cab5c1e010fd76e3c01b2aa127bbed2dbbe4f23"} Nov 02 23:14:23.185458 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:23.185442 5076 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b2ee05f445dbd781df4cb013cab5c1e010fd76e3c01b2aa127bbed2dbbe4f23" Nov 02 23:14:30.318234 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:30.318198 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b2a689-555949854b-vlxwx"] Nov 02 23:14:30.318853 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:30.318838 5076 cpu_manager.go:414] "containerMap: removing container" podUID="06e1a697-3cf2-47d6-9a09-5177da55ac16" containerName="insights-gathering" Nov 02 23:14:30.318927 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:30.318853 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06e1a697-3cf2-47d6-9a09-5177da55ac16" containerName="insights-gathering" Nov 02 23:14:30.318927 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:30.318869 5076 cpu_manager.go:414] "containerMap: removing container" podUID="eac149ce-3ee3-4232-8b82-10eb431d939d" containerName="payload-processing" Nov 02 23:14:30.318927 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:30.318874 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eac149ce-3ee3-4232-8b82-10eb431d939d" containerName="payload-processing" Nov 02 23:14:30.319027 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:30.319003 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="eac149ce-3ee3-4232-8b82-10eb431d939d" containerName="payload-processing" Nov 02 23:14:30.319027 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:30.319018 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="06e1a697-3cf2-47d6-9a09-5177da55ac16" containerName="insights-gathering" Nov 02 23:14:30.322728 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:30.322710 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2a689-555949854b-vlxwx" Nov 02 23:14:30.364897 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:30.364848 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvtf5\" (UniqueName: \"kubernetes.io/projected/8d120178-e30d-460c-a2d3-719cab71b787-kube-api-access-cvtf5\") pod \"maas-api-e2e-api-a-b2a689-555949854b-vlxwx\" (UID: \"8d120178-e30d-460c-a2d3-719cab71b787\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2a689-555949854b-vlxwx" Nov 02 23:14:30.365070 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:30.365007 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d120178-e30d-460c-a2d3-719cab71b787-openshift-service-ca\") pod \"maas-api-e2e-api-a-b2a689-555949854b-vlxwx\" (UID: \"8d120178-e30d-460c-a2d3-719cab71b787\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2a689-555949854b-vlxwx" Nov 02 23:14:30.365123 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:30.365101 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d120178-e30d-460c-a2d3-719cab71b787-metrics-certs\") pod \"maas-api-e2e-api-a-b2a689-555949854b-vlxwx\" (UID: \"8d120178-e30d-460c-a2d3-719cab71b787\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2a689-555949854b-vlxwx" Nov 02 23:14:30.365200 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:30.365181 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d120178-e30d-460c-a2d3-719cab71b787-maas-api-tls\") pod \"maas-api-e2e-api-a-b2a689-555949854b-vlxwx\" (UID: \"8d120178-e30d-460c-a2d3-719cab71b787\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2a689-555949854b-vlxwx" Nov 02 23:14:30.423768 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:30.423734 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-b2a689-serving-cert\"" Nov 02 23:14:30.427283 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:30.427261 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b2a689-555949854b-vlxwx"] Nov 02 23:14:30.467042 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:30.467007 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d120178-e30d-460c-a2d3-719cab71b787-openshift-service-ca\") pod \"maas-api-e2e-api-a-b2a689-555949854b-vlxwx\" (UID: \"8d120178-e30d-460c-a2d3-719cab71b787\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2a689-555949854b-vlxwx" Nov 02 23:14:30.467236 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:30.467179 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d120178-e30d-460c-a2d3-719cab71b787-metrics-certs\") pod \"maas-api-e2e-api-a-b2a689-555949854b-vlxwx\" (UID: \"8d120178-e30d-460c-a2d3-719cab71b787\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2a689-555949854b-vlxwx" Nov 02 23:14:30.467316 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:30.467270 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d120178-e30d-460c-a2d3-719cab71b787-maas-api-tls\") pod \"maas-api-e2e-api-a-b2a689-555949854b-vlxwx\" (UID: \"8d120178-e30d-460c-a2d3-719cab71b787\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2a689-555949854b-vlxwx" Nov 02 23:14:30.467571 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:30.467545 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvtf5\" (UniqueName: \"kubernetes.io/projected/8d120178-e30d-460c-a2d3-719cab71b787-kube-api-access-cvtf5\") pod \"maas-api-e2e-api-a-b2a689-555949854b-vlxwx\" (UID: \"8d120178-e30d-460c-a2d3-719cab71b787\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2a689-555949854b-vlxwx" Nov 02 23:14:30.467798 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:30.467780 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d120178-e30d-460c-a2d3-719cab71b787-openshift-service-ca\") pod \"maas-api-e2e-api-a-b2a689-555949854b-vlxwx\" (UID: \"8d120178-e30d-460c-a2d3-719cab71b787\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2a689-555949854b-vlxwx" Nov 02 23:14:30.468087 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:14:30.468059 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-b2a689-serving-cert: secret "maas-api-e2e-api-a-b2a689-serving-cert" not found Nov 02 23:14:30.468189 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:14:30.468134 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d120178-e30d-460c-a2d3-719cab71b787-maas-api-tls podName:8d120178-e30d-460c-a2d3-719cab71b787 nodeName:}" failed. No retries permitted until 2026-11-02 23:14:30.968115731 +0000 UTC m=+1319.936647510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8d120178-e30d-460c-a2d3-719cab71b787-maas-api-tls") pod "maas-api-e2e-api-a-b2a689-555949854b-vlxwx" (UID: "8d120178-e30d-460c-a2d3-719cab71b787") : secret "maas-api-e2e-api-a-b2a689-serving-cert" not found Nov 02 23:14:30.471500 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:30.471474 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d120178-e30d-460c-a2d3-719cab71b787-metrics-certs\") pod \"maas-api-e2e-api-a-b2a689-555949854b-vlxwx\" (UID: \"8d120178-e30d-460c-a2d3-719cab71b787\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2a689-555949854b-vlxwx" Nov 02 23:14:30.524026 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:30.523998 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvtf5\" (UniqueName: \"kubernetes.io/projected/8d120178-e30d-460c-a2d3-719cab71b787-kube-api-access-cvtf5\") pod \"maas-api-e2e-api-a-b2a689-555949854b-vlxwx\" (UID: \"8d120178-e30d-460c-a2d3-719cab71b787\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2a689-555949854b-vlxwx" Nov 02 23:14:31.019518 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:31.019467 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d120178-e30d-460c-a2d3-719cab71b787-maas-api-tls\") pod \"maas-api-e2e-api-a-b2a689-555949854b-vlxwx\" (UID: \"8d120178-e30d-460c-a2d3-719cab71b787\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2a689-555949854b-vlxwx" Nov 02 23:14:31.036423 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:31.036361 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d120178-e30d-460c-a2d3-719cab71b787-maas-api-tls\") pod \"maas-api-e2e-api-a-b2a689-555949854b-vlxwx\" (UID: \"8d120178-e30d-460c-a2d3-719cab71b787\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2a689-555949854b-vlxwx" Nov 02 23:14:31.235941 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:31.235907 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2a689-555949854b-vlxwx" Nov 02 23:14:31.614144 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:31.614112 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b2a689-555949854b-vlxwx"] Nov 02 23:14:31.615025 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:14:31.614999 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d120178_e30d_460c_a2d3_719cab71b787.slice/crio-b10fba09dcc3a45a557d4506b6296e8e560f72402a7ba44406db1d916a680c5f WatchSource:0}: Error finding container b10fba09dcc3a45a557d4506b6296e8e560f72402a7ba44406db1d916a680c5f: Status 404 returned error can't find the container with id b10fba09dcc3a45a557d4506b6296e8e560f72402a7ba44406db1d916a680c5f Nov 02 23:14:32.241411 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:32.241374 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2a689-555949854b-vlxwx" event={"ID":"8d120178-e30d-460c-a2d3-719cab71b787","Type":"ContainerStarted","Data":"b10fba09dcc3a45a557d4506b6296e8e560f72402a7ba44406db1d916a680c5f"} Nov 02 23:14:33.249685 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:33.249652 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2a689-555949854b-vlxwx" event={"ID":"8d120178-e30d-460c-a2d3-719cab71b787","Type":"ContainerStarted","Data":"968a75f8104e40c01895ead4d4d4a1bc7e535eca217c37a9e5187bfdaf3107d0"} Nov 02 23:14:33.250213 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:33.249862 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2a689-555949854b-vlxwx" Nov 02 23:14:33.364620 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:33.364567 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2a689-555949854b-vlxwx" podStartSLOduration=2.493941405 podStartE2EDuration="3.364551816s" totalImagesPullingTime="870.610411ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:14:33.362729043 +0000 UTC m=+1322.331260832" watchObservedRunningTime="2026-11-02 23:14:33.364551816 +0000 UTC m=+1322.333083637" Nov 02 23:14:39.261435 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:14:39.261403 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2a689-555949854b-vlxwx" Nov 02 23:14:39.884998 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:14:39.884954 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:15:09.884733 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:15:09.884696 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:15:18.461116 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:18.461079 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5"] Nov 02 23:15:18.466081 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:18.466048 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" Nov 02 23:15:18.525503 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:18.524762 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5"] Nov 02 23:15:18.564952 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:18.564918 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f46208-openshift-default-dockercfg-p4clx\"" Nov 02 23:15:18.633502 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:18.633466 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f19211b1-29a9-4e75-bd8a-ab654254a5a1-istio-envoy\") pod \"e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5\" (UID: \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\") " pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" Nov 02 23:15:18.633691 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:18.633527 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f19211b1-29a9-4e75-bd8a-ab654254a5a1-istio-podinfo\") pod \"e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5\" (UID: \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\") " pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" Nov 02 23:15:18.633766 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:18.633695 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f19211b1-29a9-4e75-bd8a-ab654254a5a1-istiod-ca-cert\") pod \"e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5\" (UID: \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\") " pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" Nov 02 23:15:18.633766 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:18.633738 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19211b1-29a9-4e75-bd8a-ab654254a5a1-credential-socket\") pod \"e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5\" (UID: \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\") " pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" Nov 02 23:15:18.633939 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:18.633903 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19211b1-29a9-4e75-bd8a-ab654254a5a1-workload-socket\") pod \"e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5\" (UID: \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\") " pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" Nov 02 23:15:18.634110 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:18.634084 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f19211b1-29a9-4e75-bd8a-ab654254a5a1-istio-data\") pod \"e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5\" (UID: \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\") " pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" Nov 02 23:15:18.634184 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:18.634124 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f19211b1-29a9-4e75-bd8a-ab654254a5a1-istio-token\") pod \"e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5\" (UID: \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\") " pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" Nov 02 23:15:18.634269 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:18.634245 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19211b1-29a9-4e75-bd8a-ab654254a5a1-workload-certs\") pod \"e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5\" (UID: \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\") " pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" Nov 02 23:15:18.634330 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:18.634282 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd58n\" (UniqueName: \"kubernetes.io/projected/f19211b1-29a9-4e75-bd8a-ab654254a5a1-kube-api-access-dd58n\") pod \"e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5\" (UID: \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\") " pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" Nov 02 23:15:18.736927 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:18.736807 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f19211b1-29a9-4e75-bd8a-ab654254a5a1-istio-envoy\") pod \"e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5\" (UID: \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\") " pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" Nov 02 23:15:18.737588 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:18.737558 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f19211b1-29a9-4e75-bd8a-ab654254a5a1-istio-podinfo\") pod \"e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5\" (UID: \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\") " pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" Nov 02 23:15:18.737815 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:18.737797 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f19211b1-29a9-4e75-bd8a-ab654254a5a1-istiod-ca-cert\") pod \"e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5\" (UID: \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\") " pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" Nov 02 23:15:18.745927 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:18.738652 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f19211b1-29a9-4e75-bd8a-ab654254a5a1-istiod-ca-cert\") pod \"e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5\" (UID: \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\") " pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" Nov 02 23:15:18.745927 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:18.742068 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19211b1-29a9-4e75-bd8a-ab654254a5a1-credential-socket\") pod \"e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5\" (UID: \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\") " pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" Nov 02 23:15:18.745927 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:18.742400 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19211b1-29a9-4e75-bd8a-ab654254a5a1-credential-socket\") pod \"e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5\" (UID: \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\") " pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" Nov 02 23:15:18.745927 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:18.742567 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19211b1-29a9-4e75-bd8a-ab654254a5a1-workload-socket\") pod \"e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5\" (UID: \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\") " pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" Nov 02 23:15:18.745927 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:18.742857 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19211b1-29a9-4e75-bd8a-ab654254a5a1-workload-socket\") pod \"e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5\" (UID: \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\") " pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" Nov 02 23:15:18.748938 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:18.748488 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f19211b1-29a9-4e75-bd8a-ab654254a5a1-istio-data\") pod \"e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5\" (UID: \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\") " pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" Nov 02 23:15:18.748938 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:18.748565 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f19211b1-29a9-4e75-bd8a-ab654254a5a1-istio-token\") pod \"e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5\" (UID: \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\") " pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" Nov 02 23:15:18.759932 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:18.759283 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f19211b1-29a9-4e75-bd8a-ab654254a5a1-istio-data\") pod \"e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5\" (UID: \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\") " pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" Nov 02 23:15:18.766926 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:18.761202 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19211b1-29a9-4e75-bd8a-ab654254a5a1-workload-certs\") pod \"e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5\" (UID: \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\") " pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" Nov 02 23:15:18.786025 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:18.784424 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19211b1-29a9-4e75-bd8a-ab654254a5a1-workload-certs\") pod \"e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5\" (UID: \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\") " pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" Nov 02 23:15:18.786025 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:18.784487 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd58n\" (UniqueName: \"kubernetes.io/projected/f19211b1-29a9-4e75-bd8a-ab654254a5a1-kube-api-access-dd58n\") pod \"e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5\" (UID: \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\") " pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" Nov 02 23:15:18.786810 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:18.786781 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f19211b1-29a9-4e75-bd8a-ab654254a5a1-istio-envoy\") pod \"e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5\" (UID: \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\") " pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" Nov 02 23:15:18.818940 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:18.817512 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f19211b1-29a9-4e75-bd8a-ab654254a5a1-istio-podinfo\") pod \"e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5\" (UID: \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\") " pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" Nov 02 23:15:18.821298 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:18.821065 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f19211b1-29a9-4e75-bd8a-ab654254a5a1-istio-token\") pod \"e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5\" (UID: \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\") " pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" Nov 02 23:15:18.843927 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:18.841973 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd58n\" (UniqueName: \"kubernetes.io/projected/f19211b1-29a9-4e75-bd8a-ab654254a5a1-kube-api-access-dd58n\") pod \"e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5\" (UID: \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\") " pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" Nov 02 23:15:19.083255 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:19.083165 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" Nov 02 23:15:19.505458 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:19.505429 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5"] Nov 02 23:15:19.505902 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:15:19.505865 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf19211b1_29a9_4e75_bd8a_ab654254a5a1.slice/crio-bf753d297afd9a9d2e2dce74f1147b6102d6869ecde26ad44ff6685ab2c92578 WatchSource:0}: Error finding container bf753d297afd9a9d2e2dce74f1147b6102d6869ecde26ad44ff6685ab2c92578: Status 404 returned error can't find the container with id bf753d297afd9a9d2e2dce74f1147b6102d6869ecde26ad44ff6685ab2c92578 Nov 02 23:15:19.508440 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:19.508408 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:15:19.508519 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:19.508475 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:15:19.508519 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:19.508503 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:15:19.674918 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:19.674869 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" event={"ID":"f19211b1-29a9-4e75-bd8a-ab654254a5a1","Type":"ContainerStarted","Data":"9e53c98af73a7e04c5d58dfcbcb20d3f136b8f594b9d2ecd87797920dab7c2b9"} Nov 02 23:15:19.675092 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:19.674932 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" event={"ID":"f19211b1-29a9-4e75-bd8a-ab654254a5a1","Type":"ContainerStarted","Data":"bf753d297afd9a9d2e2dce74f1147b6102d6869ecde26ad44ff6685ab2c92578"} Nov 02 23:15:19.790611 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:19.790485 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" podStartSLOduration=1.790467327 podStartE2EDuration="1.790467327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:15:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:15:19.787849083 +0000 UTC m=+1368.756380868" watchObservedRunningTime="2026-11-02 23:15:19.790467327 +0000 UTC m=+1368.758999115" Nov 02 23:15:20.084237 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:20.084144 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" Nov 02 23:15:20.102636 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:20.102603 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" Nov 02 23:15:20.682207 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:20.682174 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" Nov 02 23:15:20.683297 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:20.683277 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" Nov 02 23:15:39.884978 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:15:39.884940 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:15:44.571775 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:44.571695 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx"] Nov 02 23:15:44.577370 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:44.577350 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx" Nov 02 23:15:44.630573 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:44.630540 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx"] Nov 02 23:15:44.673745 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:44.673707 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-14b321\"/\"openshift-service-ca.crt\"" Nov 02 23:15:44.673745 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:44.673718 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-14b321\"/\"kube-root-ca.crt\"" Nov 02 23:15:44.673997 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:44.673810 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-14b321\"/\"default-dockercfg-px9j7\"" Nov 02 23:15:44.675153 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:44.675130 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-14b321\"/\"distinct-2-w3-14b321-kserve-self-signed-certs\"" Nov 02 23:15:44.762030 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:44.738856 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8r8x\" (UniqueName: \"kubernetes.io/projected/ffe70026-3a9a-49bb-bc08-e456e35a7c68-kube-api-access-s8r8x\") pod \"distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx\" (UID: \"ffe70026-3a9a-49bb-bc08-e456e35a7c68\") " pod="e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx" Nov 02 23:15:44.762030 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:44.739126 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffe70026-3a9a-49bb-bc08-e456e35a7c68-tmp-dir\") pod \"distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx\" (UID: \"ffe70026-3a9a-49bb-bc08-e456e35a7c68\") " pod="e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx" Nov 02 23:15:44.762030 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:44.739170 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffe70026-3a9a-49bb-bc08-e456e35a7c68-model-cache\") pod \"distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx\" (UID: \"ffe70026-3a9a-49bb-bc08-e456e35a7c68\") " pod="e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx" Nov 02 23:15:44.762030 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:44.739235 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffe70026-3a9a-49bb-bc08-e456e35a7c68-tls-certs\") pod \"distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx\" (UID: \"ffe70026-3a9a-49bb-bc08-e456e35a7c68\") " pod="e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx" Nov 02 23:15:44.762030 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:44.739387 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffe70026-3a9a-49bb-bc08-e456e35a7c68-dshm\") pod \"distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx\" (UID: \"ffe70026-3a9a-49bb-bc08-e456e35a7c68\") " pod="e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx" Nov 02 23:15:44.762030 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:44.739420 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffe70026-3a9a-49bb-bc08-e456e35a7c68-home\") pod \"distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx\" (UID: \"ffe70026-3a9a-49bb-bc08-e456e35a7c68\") " pod="e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx" Nov 02 23:15:44.849955 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:44.848665 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffe70026-3a9a-49bb-bc08-e456e35a7c68-tmp-dir\") pod \"distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx\" (UID: \"ffe70026-3a9a-49bb-bc08-e456e35a7c68\") " pod="e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx" Nov 02 23:15:44.849955 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:44.848706 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffe70026-3a9a-49bb-bc08-e456e35a7c68-model-cache\") pod \"distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx\" (UID: \"ffe70026-3a9a-49bb-bc08-e456e35a7c68\") " pod="e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx" Nov 02 23:15:44.849955 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:44.848744 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffe70026-3a9a-49bb-bc08-e456e35a7c68-tls-certs\") pod \"distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx\" (UID: \"ffe70026-3a9a-49bb-bc08-e456e35a7c68\") " pod="e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx" Nov 02 23:15:44.849955 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:44.848819 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffe70026-3a9a-49bb-bc08-e456e35a7c68-dshm\") pod \"distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx\" (UID: \"ffe70026-3a9a-49bb-bc08-e456e35a7c68\") " pod="e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx" Nov 02 23:15:44.849955 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:44.848833 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffe70026-3a9a-49bb-bc08-e456e35a7c68-home\") pod \"distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx\" (UID: \"ffe70026-3a9a-49bb-bc08-e456e35a7c68\") " pod="e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx" Nov 02 23:15:44.849955 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:44.848856 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8r8x\" (UniqueName: \"kubernetes.io/projected/ffe70026-3a9a-49bb-bc08-e456e35a7c68-kube-api-access-s8r8x\") pod \"distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx\" (UID: \"ffe70026-3a9a-49bb-bc08-e456e35a7c68\") " pod="e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx" Nov 02 23:15:44.849955 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:44.849449 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffe70026-3a9a-49bb-bc08-e456e35a7c68-tmp-dir\") pod \"distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx\" (UID: \"ffe70026-3a9a-49bb-bc08-e456e35a7c68\") " pod="e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx" Nov 02 23:15:44.849955 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:44.849613 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffe70026-3a9a-49bb-bc08-e456e35a7c68-model-cache\") pod \"distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx\" (UID: \"ffe70026-3a9a-49bb-bc08-e456e35a7c68\") " pod="e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx" Nov 02 23:15:44.850478 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:44.850287 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffe70026-3a9a-49bb-bc08-e456e35a7c68-home\") pod \"distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx\" (UID: \"ffe70026-3a9a-49bb-bc08-e456e35a7c68\") " pod="e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx" Nov 02 23:15:44.852246 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:44.852166 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffe70026-3a9a-49bb-bc08-e456e35a7c68-tls-certs\") pod \"distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx\" (UID: \"ffe70026-3a9a-49bb-bc08-e456e35a7c68\") " pod="e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx" Nov 02 23:15:44.852389 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:44.852292 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffe70026-3a9a-49bb-bc08-e456e35a7c68-dshm\") pod \"distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx\" (UID: \"ffe70026-3a9a-49bb-bc08-e456e35a7c68\") " pod="e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx" Nov 02 23:15:44.903376 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:44.903345 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8r8x\" (UniqueName: \"kubernetes.io/projected/ffe70026-3a9a-49bb-bc08-e456e35a7c68-kube-api-access-s8r8x\") pod \"distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx\" (UID: \"ffe70026-3a9a-49bb-bc08-e456e35a7c68\") " pod="e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx" Nov 02 23:15:45.189681 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:45.189647 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx" Nov 02 23:15:45.560289 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:45.560260 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx"] Nov 02 23:15:45.560884 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:15:45.560843 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podffe70026_3a9a_49bb_bc08_e456e35a7c68.slice/crio-b12d3ca7c3d02eba1893823a59cd947c0208daa82440bab80adf571fa8920617 WatchSource:0}: Error finding container b12d3ca7c3d02eba1893823a59cd947c0208daa82440bab80adf571fa8920617: Status 404 returned error can't find the container with id b12d3ca7c3d02eba1893823a59cd947c0208daa82440bab80adf571fa8920617 Nov 02 23:15:46.020036 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:46.019995 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx" event={"ID":"ffe70026-3a9a-49bb-bc08-e456e35a7c68","Type":"ContainerStarted","Data":"b873d663442206882a2a1a2d2a872a69057ec5dcd3ca7f1ab0df46a64d0b13fa"} Nov 02 23:15:46.020443 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:46.020047 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx" event={"ID":"ffe70026-3a9a-49bb-bc08-e456e35a7c68","Type":"ContainerStarted","Data":"b12d3ca7c3d02eba1893823a59cd947c0208daa82440bab80adf571fa8920617"} Nov 02 23:15:46.131603 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:46.131541 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx" podStartSLOduration=2.131526696 podStartE2EDuration="2.131526696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:15:46.12868805 +0000 UTC m=+1395.097219836" watchObservedRunningTime="2026-11-02 23:15:46.131526696 +0000 UTC m=+1395.100058526" Nov 02 23:15:46.190418 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:46.190386 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx" Nov 02 23:15:55.190228 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:55.190191 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx" Nov 02 23:15:55.203468 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:55.203443 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx" Nov 02 23:15:56.188810 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:15:56.188782 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx" Nov 02 23:16:01.607627 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:01.607593 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f46208-89d948c55-274ln"] Nov 02 23:16:01.614460 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:01.614428 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f46208-89d948c55-274ln" Nov 02 23:16:01.707962 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:01.707918 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6lxp\" (UniqueName: \"kubernetes.io/projected/b1d00c89-507e-49cf-b6a5-1b03044b6626-kube-api-access-s6lxp\") pod \"maas-api-e2e-api-b-f46208-89d948c55-274ln\" (UID: \"b1d00c89-507e-49cf-b6a5-1b03044b6626\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f46208-89d948c55-274ln" Nov 02 23:16:01.708135 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:01.708062 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1d00c89-507e-49cf-b6a5-1b03044b6626-metrics-certs\") pod \"maas-api-e2e-api-b-f46208-89d948c55-274ln\" (UID: \"b1d00c89-507e-49cf-b6a5-1b03044b6626\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f46208-89d948c55-274ln" Nov 02 23:16:01.708289 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:01.708264 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1d00c89-507e-49cf-b6a5-1b03044b6626-maas-api-tls\") pod \"maas-api-e2e-api-b-f46208-89d948c55-274ln\" (UID: \"b1d00c89-507e-49cf-b6a5-1b03044b6626\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f46208-89d948c55-274ln" Nov 02 23:16:01.708425 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:01.708408 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1d00c89-507e-49cf-b6a5-1b03044b6626-openshift-service-ca\") pod \"maas-api-e2e-api-b-f46208-89d948c55-274ln\" (UID: \"b1d00c89-507e-49cf-b6a5-1b03044b6626\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f46208-89d948c55-274ln" Nov 02 23:16:01.721152 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:01.721121 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-f46208-serving-cert\"" Nov 02 23:16:01.724641 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:01.724608 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f46208-89d948c55-274ln"] Nov 02 23:16:01.809622 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:01.809584 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6lxp\" (UniqueName: \"kubernetes.io/projected/b1d00c89-507e-49cf-b6a5-1b03044b6626-kube-api-access-s6lxp\") pod \"maas-api-e2e-api-b-f46208-89d948c55-274ln\" (UID: \"b1d00c89-507e-49cf-b6a5-1b03044b6626\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f46208-89d948c55-274ln" Nov 02 23:16:01.809793 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:01.809648 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1d00c89-507e-49cf-b6a5-1b03044b6626-metrics-certs\") pod \"maas-api-e2e-api-b-f46208-89d948c55-274ln\" (UID: \"b1d00c89-507e-49cf-b6a5-1b03044b6626\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f46208-89d948c55-274ln" Nov 02 23:16:01.809793 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:01.809733 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1d00c89-507e-49cf-b6a5-1b03044b6626-maas-api-tls\") pod \"maas-api-e2e-api-b-f46208-89d948c55-274ln\" (UID: \"b1d00c89-507e-49cf-b6a5-1b03044b6626\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f46208-89d948c55-274ln" Nov 02 23:16:01.809923 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:01.809807 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1d00c89-507e-49cf-b6a5-1b03044b6626-openshift-service-ca\") pod \"maas-api-e2e-api-b-f46208-89d948c55-274ln\" (UID: \"b1d00c89-507e-49cf-b6a5-1b03044b6626\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f46208-89d948c55-274ln" Nov 02 23:16:01.809923 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:16:01.809884 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-f46208-serving-cert: secret "maas-api-e2e-api-b-f46208-serving-cert" not found Nov 02 23:16:01.810048 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:16:01.809992 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1d00c89-507e-49cf-b6a5-1b03044b6626-maas-api-tls podName:b1d00c89-507e-49cf-b6a5-1b03044b6626 nodeName:}" failed. No retries permitted until 2026-11-02 23:16:02.309968051 +0000 UTC m=+1411.278499830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b1d00c89-507e-49cf-b6a5-1b03044b6626-maas-api-tls") pod "maas-api-e2e-api-b-f46208-89d948c55-274ln" (UID: "b1d00c89-507e-49cf-b6a5-1b03044b6626") : secret "maas-api-e2e-api-b-f46208-serving-cert" not found Nov 02 23:16:01.810478 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:01.810456 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1d00c89-507e-49cf-b6a5-1b03044b6626-openshift-service-ca\") pod \"maas-api-e2e-api-b-f46208-89d948c55-274ln\" (UID: \"b1d00c89-507e-49cf-b6a5-1b03044b6626\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f46208-89d948c55-274ln" Nov 02 23:16:01.812513 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:01.812487 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1d00c89-507e-49cf-b6a5-1b03044b6626-metrics-certs\") pod \"maas-api-e2e-api-b-f46208-89d948c55-274ln\" (UID: \"b1d00c89-507e-49cf-b6a5-1b03044b6626\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f46208-89d948c55-274ln" Nov 02 23:16:01.868516 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:01.868439 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6lxp\" (UniqueName: \"kubernetes.io/projected/b1d00c89-507e-49cf-b6a5-1b03044b6626-kube-api-access-s6lxp\") pod \"maas-api-e2e-api-b-f46208-89d948c55-274ln\" (UID: \"b1d00c89-507e-49cf-b6a5-1b03044b6626\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f46208-89d948c55-274ln" Nov 02 23:16:02.329998 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:02.329961 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1d00c89-507e-49cf-b6a5-1b03044b6626-maas-api-tls\") pod \"maas-api-e2e-api-b-f46208-89d948c55-274ln\" (UID: \"b1d00c89-507e-49cf-b6a5-1b03044b6626\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f46208-89d948c55-274ln" Nov 02 23:16:02.332452 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:02.332427 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1d00c89-507e-49cf-b6a5-1b03044b6626-maas-api-tls\") pod \"maas-api-e2e-api-b-f46208-89d948c55-274ln\" (UID: \"b1d00c89-507e-49cf-b6a5-1b03044b6626\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f46208-89d948c55-274ln" Nov 02 23:16:02.530978 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:02.530944 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f46208-89d948c55-274ln" Nov 02 23:16:02.901517 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:02.901487 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f46208-89d948c55-274ln"] Nov 02 23:16:02.902187 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:16:02.902161 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1d00c89_507e_49cf_b6a5_1b03044b6626.slice/crio-2a7866ea8721213a53517553ac8485e288038fadea4ea590d1f999ad2f9fa5b9 WatchSource:0}: Error finding container 2a7866ea8721213a53517553ac8485e288038fadea4ea590d1f999ad2f9fa5b9: Status 404 returned error can't find the container with id 2a7866ea8721213a53517553ac8485e288038fadea4ea590d1f999ad2f9fa5b9 Nov 02 23:16:03.254686 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:03.254654 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f46208-89d948c55-274ln" event={"ID":"b1d00c89-507e-49cf-b6a5-1b03044b6626","Type":"ContainerStarted","Data":"2a7866ea8721213a53517553ac8485e288038fadea4ea590d1f999ad2f9fa5b9"} Nov 02 23:16:04.267620 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:04.267537 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f46208-89d948c55-274ln" event={"ID":"b1d00c89-507e-49cf-b6a5-1b03044b6626","Type":"ContainerStarted","Data":"9dc063970b93656cc5e0cfd700b7b781d5b07fd7a0ad23bb5f7bac8a5f4cb451"} Nov 02 23:16:04.268082 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:04.267644 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f46208-89d948c55-274ln" Nov 02 23:16:04.379959 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:04.379905 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f46208-89d948c55-274ln" podStartSLOduration=2.294974232 podStartE2EDuration="3.37987981s" totalImagesPullingTime="1.084905578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:16:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:16:04.376800983 +0000 UTC m=+1413.345332769" watchObservedRunningTime="2026-11-02 23:16:04.37987981 +0000 UTC m=+1413.348411594" Nov 02 23:16:09.884986 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:16:09.884943 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:16:10.279956 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:10.279916 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f46208-89d948c55-274ln" Nov 02 23:16:12.684599 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:12.684567 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh"] Nov 02 23:16:12.689184 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:12.689161 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" Nov 02 23:16:12.718510 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:12.718461 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-istio-token\") pod \"e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh\" (UID: \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\") " pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" Nov 02 23:16:12.718700 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:12.718527 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-istio-data\") pod \"e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh\" (UID: \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\") " pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" Nov 02 23:16:12.718700 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:12.718608 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-istiod-ca-cert\") pod \"e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh\" (UID: \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\") " pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" Nov 02 23:16:12.718700 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:12.718659 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-workload-certs\") pod \"e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh\" (UID: \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\") " pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" Nov 02 23:16:12.718952 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:12.718723 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-istio-envoy\") pod \"e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh\" (UID: \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\") " pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" Nov 02 23:16:12.718952 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:12.718811 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-credential-socket\") pod \"e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh\" (UID: \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\") " pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" Nov 02 23:16:12.718952 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:12.718912 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-istio-podinfo\") pod \"e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh\" (UID: \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\") " pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" Nov 02 23:16:12.719128 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:12.718971 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk4db\" (UniqueName: \"kubernetes.io/projected/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-kube-api-access-kk4db\") pod \"e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh\" (UID: \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\") " pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" Nov 02 23:16:12.719128 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:12.719020 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-workload-socket\") pod \"e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh\" (UID: \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\") " pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" Nov 02 23:16:13.151990 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:13.151865 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ede388-openshift-default-dockercfg-ftgxt\"" Nov 02 23:16:13.163477 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:13.163433 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-istio-podinfo\") pod \"e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh\" (UID: \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\") " pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" Nov 02 23:16:13.202020 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:13.194959 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-istio-podinfo\") pod \"e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh\" (UID: \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\") " pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" Nov 02 23:16:13.202612 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:13.202585 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk4db\" (UniqueName: \"kubernetes.io/projected/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-kube-api-access-kk4db\") pod \"e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh\" (UID: \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\") " pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" Nov 02 23:16:13.202866 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:13.202760 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-workload-socket\") pod \"e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh\" (UID: \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\") " pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" Nov 02 23:16:13.203144 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:13.203125 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-istio-token\") pod \"e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh\" (UID: \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\") " pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" Nov 02 23:16:13.203277 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:13.203261 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-istio-data\") pod \"e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh\" (UID: \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\") " pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" Nov 02 23:16:13.203521 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:13.203480 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-istiod-ca-cert\") pod \"e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh\" (UID: \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\") " pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" Nov 02 23:16:13.203599 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:13.203567 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-workload-certs\") pod \"e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh\" (UID: \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\") " pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" Nov 02 23:16:13.203658 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:13.203647 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-istio-envoy\") pod \"e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh\" (UID: \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\") " pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" Nov 02 23:16:13.203779 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:13.203760 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-credential-socket\") pod \"e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh\" (UID: \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\") " pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" Nov 02 23:16:13.204553 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:13.204232 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-credential-socket\") pod \"e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh\" (UID: \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\") " pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" Nov 02 23:16:13.204553 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:13.204518 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-workload-socket\") pod \"e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh\" (UID: \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\") " pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" Nov 02 23:16:13.204926 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:13.204874 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-workload-certs\") pod \"e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh\" (UID: \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\") " pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" Nov 02 23:16:13.205083 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:13.205061 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-istio-data\") pod \"e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh\" (UID: \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\") " pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" Nov 02 23:16:13.205366 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:13.205347 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-istiod-ca-cert\") pod \"e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh\" (UID: \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\") " pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" Nov 02 23:16:13.207148 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:13.207121 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-istio-envoy\") pod \"e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh\" (UID: \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\") " pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" Nov 02 23:16:13.212876 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:13.212854 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh"] Nov 02 23:16:13.257165 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:13.257128 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk4db\" (UniqueName: \"kubernetes.io/projected/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-kube-api-access-kk4db\") pod \"e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh\" (UID: \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\") " pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" Nov 02 23:16:13.257456 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:13.257434 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-istio-token\") pod \"e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh\" (UID: \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\") " pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" Nov 02 23:16:13.303277 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:13.303245 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" Nov 02 23:16:13.724947 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:16:13.724914 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod818ebc94_d373_4c81_8c1f_d8a1a30bcbb1.slice/crio-a04f5f8d22eca270fa6adca92f48ef8116346bc439db87e76863ecf7bbdf4a37 WatchSource:0}: Error finding container a04f5f8d22eca270fa6adca92f48ef8116346bc439db87e76863ecf7bbdf4a37: Status 404 returned error can't find the container with id a04f5f8d22eca270fa6adca92f48ef8116346bc439db87e76863ecf7bbdf4a37 Nov 02 23:16:13.725389 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:13.725053 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh"] Nov 02 23:16:13.727412 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:13.727374 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:16:13.727514 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:13.727437 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:16:13.727514 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:13.727466 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:16:14.333402 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:14.333366 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" event={"ID":"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1","Type":"ContainerStarted","Data":"ea9410cc6a4527d30d9e68d51b4abef2975fd9e8243fc7b05c70951d7fbe499e"} Nov 02 23:16:14.333602 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:14.333417 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" event={"ID":"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1","Type":"ContainerStarted","Data":"a04f5f8d22eca270fa6adca92f48ef8116346bc439db87e76863ecf7bbdf4a37"} Nov 02 23:16:14.456341 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:14.456286 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" podStartSLOduration=2.4562705400000002 podStartE2EDuration="2.45627054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:16:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:16:14.45358116 +0000 UTC m=+1423.422112946" watchObservedRunningTime="2026-11-02 23:16:14.45627054 +0000 UTC m=+1423.424802324" Nov 02 23:16:15.303856 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:15.303827 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" Nov 02 23:16:15.308625 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:15.308600 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" Nov 02 23:16:15.342866 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:15.342370 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" Nov 02 23:16:15.343332 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:15.343306 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" Nov 02 23:16:26.212400 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:26.211980 5076 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx" podUID="ffe70026-3a9a-49bb-bc08-e456e35a7c68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:16:26.253458 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:26.253055 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx" podUID="ffe70026-3a9a-49bb-bc08-e456e35a7c68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:16:26.543971 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:26.537108 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:16:26.543971 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:26.537180 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" podUID="db841b5b-fade-48e4-9515-ef923b888f42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:16:34.606870 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:34.606834 5076 generic.go:356] "Generic (PLEG): container finished" podID="f19211b1-29a9-4e75-bd8a-ab654254a5a1" containerID="9e53c98af73a7e04c5d58dfcbcb20d3f136b8f594b9d2ecd87797920dab7c2b9" exitCode=0 Nov 02 23:16:34.607427 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:34.606939 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" event={"ID":"f19211b1-29a9-4e75-bd8a-ab654254a5a1","Type":"ContainerDied","Data":"9e53c98af73a7e04c5d58dfcbcb20d3f136b8f594b9d2ecd87797920dab7c2b9"} Nov 02 23:16:34.607522 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:34.607495 5076 scope.go:122] "RemoveContainer" containerID="9e53c98af73a7e04c5d58dfcbcb20d3f136b8f594b9d2ecd87797920dab7c2b9" Nov 02 23:16:34.609065 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:34.609035 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:16:34.609156 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:34.609100 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:16:34.609156 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:34.609131 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:16:35.639264 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:35.639230 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" event={"ID":"f19211b1-29a9-4e75-bd8a-ab654254a5a1","Type":"ContainerStarted","Data":"6e7d7ee86caa0d0a1f695f502d590ab2f7b8913972325d81589d247a5d059ad5"} Nov 02 23:16:35.682591 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:35.682563 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" Nov 02 23:16:36.083971 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:36.083937 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" Nov 02 23:16:36.085141 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:36.085112 5076 patch_prober.go:29] interesting pod/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 02 23:16:36.085277 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:36.085161 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" podUID="f19211b1-29a9-4e75-bd8a-ab654254a5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 02 23:16:37.084158 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:37.084111 5076 patch_prober.go:29] interesting pod/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 02 23:16:37.084653 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:37.084192 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" podUID="f19211b1-29a9-4e75-bd8a-ab654254a5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 02 23:16:38.084667 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:38.084616 5076 patch_prober.go:29] interesting pod/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 02 23:16:38.085182 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:38.084694 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" podUID="f19211b1-29a9-4e75-bd8a-ab654254a5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 02 23:16:39.083534 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:39.083484 5076 patch_prober.go:29] interesting pod/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 02 23:16:39.083731 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:39.083563 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" podUID="f19211b1-29a9-4e75-bd8a-ab654254a5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 02 23:16:39.884938 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:16:39.884868 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:16:40.084499 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:40.084468 5076 patch_prober.go:29] interesting pod/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 02 23:16:40.084705 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:40.084538 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" podUID="f19211b1-29a9-4e75-bd8a-ab654254a5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 02 23:16:41.088615 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:41.088581 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" Nov 02 23:16:41.089741 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:41.089718 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" Nov 02 23:16:55.692447 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:16:55.692387 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vl8q5"] Nov 02 23:17:00.694448 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:00.694384 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vl8q5" podUID="6c00852d-1d5f-4caf-9df9-78c76b570a77" containerName="payload-pre-processing" containerID="cri-o://dcbc448dbabf8800a65ceee3c9783f45ee21a2b610f769f85f787275f444d8dc" gracePeriod=25 Nov 02 23:17:03.190638 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:03.190604 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ede388-667f44657f-5hr7f"] Nov 02 23:17:03.194086 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:03.194064 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ede388-667f44657f-5hr7f" Nov 02 23:17:03.244034 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:03.244002 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ede388-serving-cert\"" Nov 02 23:17:03.249949 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:03.249878 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b77a4e24-71dc-49c2-8832-f64c72e0e54a-openshift-service-ca\") pod \"maas-api-e2e-api-a-ede388-667f44657f-5hr7f\" (UID: \"b77a4e24-71dc-49c2-8832-f64c72e0e54a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ede388-667f44657f-5hr7f" Nov 02 23:17:03.250212 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:03.250185 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b77a4e24-71dc-49c2-8832-f64c72e0e54a-maas-api-tls\") pod \"maas-api-e2e-api-a-ede388-667f44657f-5hr7f\" (UID: \"b77a4e24-71dc-49c2-8832-f64c72e0e54a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ede388-667f44657f-5hr7f" Nov 02 23:17:03.250408 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:03.250388 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgf5r\" (UniqueName: \"kubernetes.io/projected/b77a4e24-71dc-49c2-8832-f64c72e0e54a-kube-api-access-kgf5r\") pod \"maas-api-e2e-api-a-ede388-667f44657f-5hr7f\" (UID: \"b77a4e24-71dc-49c2-8832-f64c72e0e54a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ede388-667f44657f-5hr7f" Nov 02 23:17:03.250514 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:03.250496 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b77a4e24-71dc-49c2-8832-f64c72e0e54a-metrics-certs\") pod \"maas-api-e2e-api-a-ede388-667f44657f-5hr7f\" (UID: \"b77a4e24-71dc-49c2-8832-f64c72e0e54a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ede388-667f44657f-5hr7f" Nov 02 23:17:03.297389 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:03.297350 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ede388-667f44657f-5hr7f"] Nov 02 23:17:03.352492 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:03.352455 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgf5r\" (UniqueName: \"kubernetes.io/projected/b77a4e24-71dc-49c2-8832-f64c72e0e54a-kube-api-access-kgf5r\") pod \"maas-api-e2e-api-a-ede388-667f44657f-5hr7f\" (UID: \"b77a4e24-71dc-49c2-8832-f64c72e0e54a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ede388-667f44657f-5hr7f" Nov 02 23:17:03.352699 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:03.352527 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b77a4e24-71dc-49c2-8832-f64c72e0e54a-metrics-certs\") pod \"maas-api-e2e-api-a-ede388-667f44657f-5hr7f\" (UID: \"b77a4e24-71dc-49c2-8832-f64c72e0e54a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ede388-667f44657f-5hr7f" Nov 02 23:17:03.352699 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:03.352579 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b77a4e24-71dc-49c2-8832-f64c72e0e54a-openshift-service-ca\") pod \"maas-api-e2e-api-a-ede388-667f44657f-5hr7f\" (UID: \"b77a4e24-71dc-49c2-8832-f64c72e0e54a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ede388-667f44657f-5hr7f" Nov 02 23:17:03.352818 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:03.352780 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b77a4e24-71dc-49c2-8832-f64c72e0e54a-maas-api-tls\") pod \"maas-api-e2e-api-a-ede388-667f44657f-5hr7f\" (UID: \"b77a4e24-71dc-49c2-8832-f64c72e0e54a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ede388-667f44657f-5hr7f" Nov 02 23:17:03.353482 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:17:03.353454 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-ede388-serving-cert: secret "maas-api-e2e-api-a-ede388-serving-cert" not found Nov 02 23:17:03.353608 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:17:03.353538 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b77a4e24-71dc-49c2-8832-f64c72e0e54a-maas-api-tls podName:b77a4e24-71dc-49c2-8832-f64c72e0e54a nodeName:}" failed. No retries permitted until 2026-11-02 23:17:03.853519588 +0000 UTC m=+1472.822051364 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b77a4e24-71dc-49c2-8832-f64c72e0e54a-maas-api-tls") pod "maas-api-e2e-api-a-ede388-667f44657f-5hr7f" (UID: "b77a4e24-71dc-49c2-8832-f64c72e0e54a") : secret "maas-api-e2e-api-a-ede388-serving-cert" not found Nov 02 23:17:03.353694 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:03.353617 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b77a4e24-71dc-49c2-8832-f64c72e0e54a-openshift-service-ca\") pod \"maas-api-e2e-api-a-ede388-667f44657f-5hr7f\" (UID: \"b77a4e24-71dc-49c2-8832-f64c72e0e54a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ede388-667f44657f-5hr7f" Nov 02 23:17:03.356434 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:03.356399 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b77a4e24-71dc-49c2-8832-f64c72e0e54a-metrics-certs\") pod \"maas-api-e2e-api-a-ede388-667f44657f-5hr7f\" (UID: \"b77a4e24-71dc-49c2-8832-f64c72e0e54a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ede388-667f44657f-5hr7f" Nov 02 23:17:03.408114 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:03.408078 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgf5r\" (UniqueName: \"kubernetes.io/projected/b77a4e24-71dc-49c2-8832-f64c72e0e54a-kube-api-access-kgf5r\") pod \"maas-api-e2e-api-a-ede388-667f44657f-5hr7f\" (UID: \"b77a4e24-71dc-49c2-8832-f64c72e0e54a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ede388-667f44657f-5hr7f" Nov 02 23:17:03.630818 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:03.630710 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vl8q5" podUID="6c00852d-1d5f-4caf-9df9-78c76b570a77" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 23:17:03.870078 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:03.870040 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b77a4e24-71dc-49c2-8832-f64c72e0e54a-maas-api-tls\") pod \"maas-api-e2e-api-a-ede388-667f44657f-5hr7f\" (UID: \"b77a4e24-71dc-49c2-8832-f64c72e0e54a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ede388-667f44657f-5hr7f" Nov 02 23:17:03.872579 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:03.872550 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b77a4e24-71dc-49c2-8832-f64c72e0e54a-maas-api-tls\") pod \"maas-api-e2e-api-a-ede388-667f44657f-5hr7f\" (UID: \"b77a4e24-71dc-49c2-8832-f64c72e0e54a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ede388-667f44657f-5hr7f" Nov 02 23:17:04.108220 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:04.108184 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ede388-667f44657f-5hr7f" Nov 02 23:17:04.484179 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:04.484147 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ede388-667f44657f-5hr7f"] Nov 02 23:17:04.485265 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:17:04.485237 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb77a4e24_71dc_49c2_8832_f64c72e0e54a.slice/crio-109d55f277d7dcb2062e4bb494f8d40b4336eb775558c3daf458cb0d97d74c88 WatchSource:0}: Error finding container 109d55f277d7dcb2062e4bb494f8d40b4336eb775558c3daf458cb0d97d74c88: Status 404 returned error can't find the container with id 109d55f277d7dcb2062e4bb494f8d40b4336eb775558c3daf458cb0d97d74c88 Nov 02 23:17:04.850538 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:04.850438 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ede388-667f44657f-5hr7f" event={"ID":"b77a4e24-71dc-49c2-8832-f64c72e0e54a","Type":"ContainerStarted","Data":"109d55f277d7dcb2062e4bb494f8d40b4336eb775558c3daf458cb0d97d74c88"} Nov 02 23:17:05.860531 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:05.860489 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ede388-667f44657f-5hr7f" event={"ID":"b77a4e24-71dc-49c2-8832-f64c72e0e54a","Type":"ContainerStarted","Data":"13064b3020a8cb56457c9c729b47379a35bfb6e94176af55641e8ee0f8391938"} Nov 02 23:17:05.861170 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:05.861149 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ede388-667f44657f-5hr7f" Nov 02 23:17:05.975691 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:05.975624 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ede388-667f44657f-5hr7f" podStartSLOduration=2.060256265 podStartE2EDuration="2.975609965s" totalImagesPullingTime="915.3537ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:17:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:17:05.972321323 +0000 UTC m=+1474.940853108" watchObservedRunningTime="2026-11-02 23:17:05.975609965 +0000 UTC m=+1474.944141749" Nov 02 23:17:09.884216 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:17:09.884176 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:17:12.880272 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:12.880198 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ede388-667f44657f-5hr7f" Nov 02 23:17:13.630844 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:13.630806 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vl8q5" podUID="6c00852d-1d5f-4caf-9df9-78c76b570a77" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 23:17:20.983804 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:20.983777 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vl8q5" Nov 02 23:17:21.003503 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:21.003465 5076 generic.go:356] "Generic (PLEG): container finished" podID="6c00852d-1d5f-4caf-9df9-78c76b570a77" containerID="dcbc448dbabf8800a65ceee3c9783f45ee21a2b610f769f85f787275f444d8dc" exitCode=0 Nov 02 23:17:21.003679 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:21.003514 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vl8q5" Nov 02 23:17:21.003679 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:21.003563 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vl8q5" event={"ID":"6c00852d-1d5f-4caf-9df9-78c76b570a77","Type":"ContainerDied","Data":"dcbc448dbabf8800a65ceee3c9783f45ee21a2b610f769f85f787275f444d8dc"} Nov 02 23:17:21.003679 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:21.003636 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vl8q5" event={"ID":"6c00852d-1d5f-4caf-9df9-78c76b570a77","Type":"ContainerDied","Data":"b869936754d158cae930bedc183c9d38d53e500e13ad5d37070f4f999a789ea6"} Nov 02 23:17:21.003679 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:21.003668 5076 scope.go:122] "RemoveContainer" containerID="dcbc448dbabf8800a65ceee3c9783f45ee21a2b610f769f85f787275f444d8dc" Nov 02 23:17:21.018166 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:21.018141 5076 scope.go:122] "RemoveContainer" containerID="dcbc448dbabf8800a65ceee3c9783f45ee21a2b610f769f85f787275f444d8dc" Nov 02 23:17:21.018509 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:17:21.018479 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcbc448dbabf8800a65ceee3c9783f45ee21a2b610f769f85f787275f444d8dc\": container with ID starting with dcbc448dbabf8800a65ceee3c9783f45ee21a2b610f769f85f787275f444d8dc not found: ID does not exist" containerID="dcbc448dbabf8800a65ceee3c9783f45ee21a2b610f769f85f787275f444d8dc" Nov 02 23:17:21.018619 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:21.018519 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcbc448dbabf8800a65ceee3c9783f45ee21a2b610f769f85f787275f444d8dc"} err="failed to get container status \"dcbc448dbabf8800a65ceee3c9783f45ee21a2b610f769f85f787275f444d8dc\": rpc error: code = NotFound desc = could not find container \"dcbc448dbabf8800a65ceee3c9783f45ee21a2b610f769f85f787275f444d8dc\": container with ID starting with dcbc448dbabf8800a65ceee3c9783f45ee21a2b610f769f85f787275f444d8dc not found: ID does not exist" Nov 02 23:17:21.073111 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:21.073083 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c00852d-1d5f-4caf-9df9-78c76b570a77-kube-api-access-86wn7\" (UniqueName: \"kubernetes.io/projected/6c00852d-1d5f-4caf-9df9-78c76b570a77-kube-api-access-86wn7\") pod \"6c00852d-1d5f-4caf-9df9-78c76b570a77\" (UID: \"6c00852d-1d5f-4caf-9df9-78c76b570a77\") " Nov 02 23:17:21.073281 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:21.073242 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c00852d-1d5f-4caf-9df9-78c76b570a77-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c00852d-1d5f-4caf-9df9-78c76b570a77-plugins-config-volume\") pod \"6c00852d-1d5f-4caf-9df9-78c76b570a77\" (UID: \"6c00852d-1d5f-4caf-9df9-78c76b570a77\") " Nov 02 23:17:21.073608 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:21.073583 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c00852d-1d5f-4caf-9df9-78c76b570a77-plugins-config-volume" pod "6c00852d-1d5f-4caf-9df9-78c76b570a77" (UID: "6c00852d-1d5f-4caf-9df9-78c76b570a77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:17:21.073983 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:21.073960 5076 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c00852d-1d5f-4caf-9df9-78c76b570a77-plugins-config-volume\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:17:21.075406 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:21.075385 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c00852d-1d5f-4caf-9df9-78c76b570a77-kube-api-access-86wn7" pod "6c00852d-1d5f-4caf-9df9-78c76b570a77" (UID: "6c00852d-1d5f-4caf-9df9-78c76b570a77"). InnerVolumeSpecName "kube-api-access-86wn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:17:21.175732 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:21.175702 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86wn7\" (UniqueName: \"kubernetes.io/projected/6c00852d-1d5f-4caf-9df9-78c76b570a77-kube-api-access-86wn7\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:17:21.414097 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:21.414005 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vl8q5"] Nov 02 23:17:21.417290 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:21.417263 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vl8q5"] Nov 02 23:17:22.917305 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:22.917271 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b2a689-555949854b-vlxwx"] Nov 02 23:17:22.917724 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:22.917504 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2a689-555949854b-vlxwx" podUID="8d120178-e30d-460c-a2d3-719cab71b787" containerName="maas-api" containerID="cri-o://968a75f8104e40c01895ead4d4d4a1bc7e535eca217c37a9e5187bfdaf3107d0" gracePeriod=30 Nov 02 23:17:23.287135 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:23.287091 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c00852d-1d5f-4caf-9df9-78c76b570a77" path="/var/lib/kubelet/pods/6c00852d-1d5f-4caf-9df9-78c76b570a77/volumes" Nov 02 23:17:23.317914 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:23.312756 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2a689-555949854b-vlxwx" Nov 02 23:17:23.431611 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:23.431570 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d120178-e30d-460c-a2d3-719cab71b787-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d120178-e30d-460c-a2d3-719cab71b787-metrics-certs\") pod \"8d120178-e30d-460c-a2d3-719cab71b787\" (UID: \"8d120178-e30d-460c-a2d3-719cab71b787\") " Nov 02 23:17:23.432151 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:23.432123 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d120178-e30d-460c-a2d3-719cab71b787-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d120178-e30d-460c-a2d3-719cab71b787-openshift-service-ca\") pod \"8d120178-e30d-460c-a2d3-719cab71b787\" (UID: \"8d120178-e30d-460c-a2d3-719cab71b787\") " Nov 02 23:17:23.432382 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:23.432366 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d120178-e30d-460c-a2d3-719cab71b787-kube-api-access-cvtf5\" (UniqueName: \"kubernetes.io/projected/8d120178-e30d-460c-a2d3-719cab71b787-kube-api-access-cvtf5\") pod \"8d120178-e30d-460c-a2d3-719cab71b787\" (UID: \"8d120178-e30d-460c-a2d3-719cab71b787\") " Nov 02 23:17:23.432515 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:23.432501 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d120178-e30d-460c-a2d3-719cab71b787-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d120178-e30d-460c-a2d3-719cab71b787-maas-api-tls\") pod \"8d120178-e30d-460c-a2d3-719cab71b787\" (UID: \"8d120178-e30d-460c-a2d3-719cab71b787\") " Nov 02 23:17:23.433925 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:23.433883 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d120178-e30d-460c-a2d3-719cab71b787-openshift-service-ca" pod "8d120178-e30d-460c-a2d3-719cab71b787" (UID: "8d120178-e30d-460c-a2d3-719cab71b787"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:17:23.469545 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:23.469462 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d120178-e30d-460c-a2d3-719cab71b787-kube-api-access-cvtf5" pod "8d120178-e30d-460c-a2d3-719cab71b787" (UID: "8d120178-e30d-460c-a2d3-719cab71b787"). InnerVolumeSpecName "kube-api-access-cvtf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:17:23.470689 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:23.470648 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d120178-e30d-460c-a2d3-719cab71b787-metrics-certs" pod "8d120178-e30d-460c-a2d3-719cab71b787" (UID: "8d120178-e30d-460c-a2d3-719cab71b787"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:17:23.470958 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:23.470939 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d120178-e30d-460c-a2d3-719cab71b787-maas-api-tls" pod "8d120178-e30d-460c-a2d3-719cab71b787" (UID: "8d120178-e30d-460c-a2d3-719cab71b787"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:17:23.536778 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:23.536728 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d120178-e30d-460c-a2d3-719cab71b787-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:17:23.537048 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:23.537019 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvtf5\" (UniqueName: \"kubernetes.io/projected/8d120178-e30d-460c-a2d3-719cab71b787-kube-api-access-cvtf5\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:17:23.537048 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:23.537049 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d120178-e30d-460c-a2d3-719cab71b787-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:17:23.537201 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:23.537062 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d120178-e30d-460c-a2d3-719cab71b787-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:17:24.037925 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:24.035306 5076 generic.go:356] "Generic (PLEG): container finished" podID="8d120178-e30d-460c-a2d3-719cab71b787" containerID="968a75f8104e40c01895ead4d4d4a1bc7e535eca217c37a9e5187bfdaf3107d0" exitCode=0 Nov 02 23:17:24.037925 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:24.035413 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2a689-555949854b-vlxwx" event={"ID":"8d120178-e30d-460c-a2d3-719cab71b787","Type":"ContainerDied","Data":"968a75f8104e40c01895ead4d4d4a1bc7e535eca217c37a9e5187bfdaf3107d0"} Nov 02 23:17:24.037925 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:24.035502 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2a689-555949854b-vlxwx" event={"ID":"8d120178-e30d-460c-a2d3-719cab71b787","Type":"ContainerDied","Data":"b10fba09dcc3a45a557d4506b6296e8e560f72402a7ba44406db1d916a680c5f"} Nov 02 23:17:24.037925 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:24.035543 5076 scope.go:122] "RemoveContainer" containerID="968a75f8104e40c01895ead4d4d4a1bc7e535eca217c37a9e5187bfdaf3107d0" Nov 02 23:17:24.037925 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:24.035705 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2a689-555949854b-vlxwx" Nov 02 23:17:24.126276 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:24.126247 5076 scope.go:122] "RemoveContainer" containerID="968a75f8104e40c01895ead4d4d4a1bc7e535eca217c37a9e5187bfdaf3107d0" Nov 02 23:17:24.126806 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:17:24.126786 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"968a75f8104e40c01895ead4d4d4a1bc7e535eca217c37a9e5187bfdaf3107d0\": container with ID starting with 968a75f8104e40c01895ead4d4d4a1bc7e535eca217c37a9e5187bfdaf3107d0 not found: ID does not exist" containerID="968a75f8104e40c01895ead4d4d4a1bc7e535eca217c37a9e5187bfdaf3107d0" Nov 02 23:17:24.127347 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:24.127309 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"968a75f8104e40c01895ead4d4d4a1bc7e535eca217c37a9e5187bfdaf3107d0"} err="failed to get container status \"968a75f8104e40c01895ead4d4d4a1bc7e535eca217c37a9e5187bfdaf3107d0\": rpc error: code = NotFound desc = could not find container \"968a75f8104e40c01895ead4d4d4a1bc7e535eca217c37a9e5187bfdaf3107d0\": container with ID starting with 968a75f8104e40c01895ead4d4d4a1bc7e535eca217c37a9e5187bfdaf3107d0 not found: ID does not exist" Nov 02 23:17:24.241355 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:24.241320 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b2a689-555949854b-vlxwx"] Nov 02 23:17:24.288256 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:24.288174 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b2a689-555949854b-vlxwx"] Nov 02 23:17:25.148686 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:25.148655 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d120178-e30d-460c-a2d3-719cab71b787" path="/var/lib/kubelet/pods/8d120178-e30d-460c-a2d3-719cab71b787/volumes" Nov 02 23:17:29.197937 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:29.197868 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq"] Nov 02 23:17:29.198523 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:29.198509 5076 cpu_manager.go:414] "containerMap: removing container" podUID="8d120178-e30d-460c-a2d3-719cab71b787" containerName="maas-api" Nov 02 23:17:29.198567 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:29.198524 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d120178-e30d-460c-a2d3-719cab71b787" containerName="maas-api" Nov 02 23:17:29.198567 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:29.198562 5076 cpu_manager.go:414] "containerMap: removing container" podUID="6c00852d-1d5f-4caf-9df9-78c76b570a77" containerName="payload-pre-processing" Nov 02 23:17:29.198635 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:29.198567 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c00852d-1d5f-4caf-9df9-78c76b570a77" containerName="payload-pre-processing" Nov 02 23:17:29.198672 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:29.198652 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c00852d-1d5f-4caf-9df9-78c76b570a77" containerName="payload-pre-processing" Nov 02 23:17:29.198672 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:29.198664 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d120178-e30d-460c-a2d3-719cab71b787" containerName="maas-api" Nov 02 23:17:29.201842 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:29.201823 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" Nov 02 23:17:29.379913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:29.375136 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-68e067-openshift-default-dockercfg-x9mhl\"" Nov 02 23:17:29.403915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:29.400609 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e055ddbc-25c2-439e-85a6-b16c23688888-istio-token\") pod \"e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq\" (UID: \"e055ddbc-25c2-439e-85a6-b16c23688888\") " pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" Nov 02 23:17:29.403915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:29.400659 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e055ddbc-25c2-439e-85a6-b16c23688888-workload-certs\") pod \"e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq\" (UID: \"e055ddbc-25c2-439e-85a6-b16c23688888\") " pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" Nov 02 23:17:29.403915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:29.400713 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e055ddbc-25c2-439e-85a6-b16c23688888-istio-data\") pod \"e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq\" (UID: \"e055ddbc-25c2-439e-85a6-b16c23688888\") " pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" Nov 02 23:17:29.403915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:29.400738 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e055ddbc-25c2-439e-85a6-b16c23688888-istiod-ca-cert\") pod \"e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq\" (UID: \"e055ddbc-25c2-439e-85a6-b16c23688888\") " pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" Nov 02 23:17:29.403915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:29.400759 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e055ddbc-25c2-439e-85a6-b16c23688888-istio-podinfo\") pod \"e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq\" (UID: \"e055ddbc-25c2-439e-85a6-b16c23688888\") " pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" Nov 02 23:17:29.403915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:29.400840 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e055ddbc-25c2-439e-85a6-b16c23688888-istio-envoy\") pod \"e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq\" (UID: \"e055ddbc-25c2-439e-85a6-b16c23688888\") " pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" Nov 02 23:17:29.427085 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:29.427049 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq"] Nov 02 23:17:29.438929 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:29.436724 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e055ddbc-25c2-439e-85a6-b16c23688888-credential-socket\") pod \"e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq\" (UID: \"e055ddbc-25c2-439e-85a6-b16c23688888\") " pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" Nov 02 23:17:29.438929 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:29.436943 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6cx8\" (UniqueName: \"kubernetes.io/projected/e055ddbc-25c2-439e-85a6-b16c23688888-kube-api-access-q6cx8\") pod \"e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq\" (UID: \"e055ddbc-25c2-439e-85a6-b16c23688888\") " pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" Nov 02 23:17:29.438929 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:29.437006 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e055ddbc-25c2-439e-85a6-b16c23688888-workload-socket\") pod \"e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq\" (UID: \"e055ddbc-25c2-439e-85a6-b16c23688888\") " pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" Nov 02 23:17:29.626541 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:29.626445 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e055ddbc-25c2-439e-85a6-b16c23688888-credential-socket\") pod \"e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq\" (UID: \"e055ddbc-25c2-439e-85a6-b16c23688888\") " pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" Nov 02 23:17:29.642914 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:29.626857 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6cx8\" (UniqueName: \"kubernetes.io/projected/e055ddbc-25c2-439e-85a6-b16c23688888-kube-api-access-q6cx8\") pod \"e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq\" (UID: \"e055ddbc-25c2-439e-85a6-b16c23688888\") " pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" Nov 02 23:17:29.643271 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:29.643237 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e055ddbc-25c2-439e-85a6-b16c23688888-workload-socket\") pod \"e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq\" (UID: \"e055ddbc-25c2-439e-85a6-b16c23688888\") " pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" Nov 02 23:17:29.647081 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:29.647049 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e055ddbc-25c2-439e-85a6-b16c23688888-istio-token\") pod \"e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq\" (UID: \"e055ddbc-25c2-439e-85a6-b16c23688888\") " pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" Nov 02 23:17:29.657116 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:29.648798 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e055ddbc-25c2-439e-85a6-b16c23688888-credential-socket\") pod \"e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq\" (UID: \"e055ddbc-25c2-439e-85a6-b16c23688888\") " pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" Nov 02 23:17:29.657392 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:29.651295 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e055ddbc-25c2-439e-85a6-b16c23688888-workload-socket\") pod \"e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq\" (UID: \"e055ddbc-25c2-439e-85a6-b16c23688888\") " pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" Nov 02 23:17:29.657973 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:29.657906 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e055ddbc-25c2-439e-85a6-b16c23688888-workload-certs\") pod \"e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq\" (UID: \"e055ddbc-25c2-439e-85a6-b16c23688888\") " pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" Nov 02 23:17:29.658172 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:29.658146 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e055ddbc-25c2-439e-85a6-b16c23688888-workload-certs\") pod \"e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq\" (UID: \"e055ddbc-25c2-439e-85a6-b16c23688888\") " pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" Nov 02 23:17:29.658374 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:29.658354 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e055ddbc-25c2-439e-85a6-b16c23688888-istio-data\") pod \"e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq\" (UID: \"e055ddbc-25c2-439e-85a6-b16c23688888\") " pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" Nov 02 23:17:29.658871 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:29.658795 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e055ddbc-25c2-439e-85a6-b16c23688888-istio-data\") pod \"e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq\" (UID: \"e055ddbc-25c2-439e-85a6-b16c23688888\") " pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" Nov 02 23:17:29.672956 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:29.672833 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e055ddbc-25c2-439e-85a6-b16c23688888-istiod-ca-cert\") pod \"e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq\" (UID: \"e055ddbc-25c2-439e-85a6-b16c23688888\") " pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" Nov 02 23:17:29.673244 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:29.658844 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e055ddbc-25c2-439e-85a6-b16c23688888-istiod-ca-cert\") pod \"e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq\" (UID: \"e055ddbc-25c2-439e-85a6-b16c23688888\") " pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" Nov 02 23:17:29.674067 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:29.673449 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e055ddbc-25c2-439e-85a6-b16c23688888-istio-podinfo\") pod \"e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq\" (UID: \"e055ddbc-25c2-439e-85a6-b16c23688888\") " pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" Nov 02 23:17:29.674307 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:29.674287 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e055ddbc-25c2-439e-85a6-b16c23688888-istio-envoy\") pod \"e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq\" (UID: \"e055ddbc-25c2-439e-85a6-b16c23688888\") " pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" Nov 02 23:17:29.691922 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:29.690728 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e055ddbc-25c2-439e-85a6-b16c23688888-istio-podinfo\") pod \"e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq\" (UID: \"e055ddbc-25c2-439e-85a6-b16c23688888\") " pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" Nov 02 23:17:29.713133 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:29.713099 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6cx8\" (UniqueName: \"kubernetes.io/projected/e055ddbc-25c2-439e-85a6-b16c23688888-kube-api-access-q6cx8\") pod \"e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq\" (UID: \"e055ddbc-25c2-439e-85a6-b16c23688888\") " pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" Nov 02 23:17:29.713461 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:29.713138 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e055ddbc-25c2-439e-85a6-b16c23688888-istio-envoy\") pod \"e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq\" (UID: \"e055ddbc-25c2-439e-85a6-b16c23688888\") " pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" Nov 02 23:17:29.715501 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:29.715478 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e055ddbc-25c2-439e-85a6-b16c23688888-istio-token\") pod \"e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq\" (UID: \"e055ddbc-25c2-439e-85a6-b16c23688888\") " pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" Nov 02 23:17:29.814724 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:29.814686 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" Nov 02 23:17:30.232362 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:30.232329 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq"] Nov 02 23:17:30.232776 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:17:30.232730 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode055ddbc_25c2_439e_85a6_b16c23688888.slice/crio-8692f8cdbbabcfbff6154c42adaa0d64e14f30d2dfbe1f892981508ab03c3883 WatchSource:0}: Error finding container 8692f8cdbbabcfbff6154c42adaa0d64e14f30d2dfbe1f892981508ab03c3883: Status 404 returned error can't find the container with id 8692f8cdbbabcfbff6154c42adaa0d64e14f30d2dfbe1f892981508ab03c3883 Nov 02 23:17:30.235441 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:30.235412 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:17:30.235522 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:30.235473 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:17:30.235522 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:30.235506 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:17:31.088053 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:31.088013 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" event={"ID":"e055ddbc-25c2-439e-85a6-b16c23688888","Type":"ContainerStarted","Data":"bb00f3e50511a389808878d36acc49d0ce70589eb29effdc2dee1cf0ce556f2e"} Nov 02 23:17:31.088229 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:31.088077 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" event={"ID":"e055ddbc-25c2-439e-85a6-b16c23688888","Type":"ContainerStarted","Data":"8692f8cdbbabcfbff6154c42adaa0d64e14f30d2dfbe1f892981508ab03c3883"} Nov 02 23:17:31.246876 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:31.246824 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" podStartSLOduration=2.246811362 podStartE2EDuration="2.246811362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:17:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:17:31.246200954 +0000 UTC m=+1500.214732763" watchObservedRunningTime="2026-11-02 23:17:31.246811362 +0000 UTC m=+1500.215343146" Nov 02 23:17:31.653363 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:31.653324 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 02 23:17:31.669675 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:31.669650 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 02 23:17:31.815199 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:31.815110 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" Nov 02 23:17:31.820281 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:31.820256 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" Nov 02 23:17:32.111969 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:32.109003 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" Nov 02 23:17:32.123679 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:32.119219 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" Nov 02 23:17:39.885122 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:17:39.885072 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:17:53.309050 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:53.309007 5076 generic.go:356] "Generic (PLEG): container finished" podID="818ebc94-d373-4c81-8c1f-d8a1a30bcbb1" containerID="ea9410cc6a4527d30d9e68d51b4abef2975fd9e8243fc7b05c70951d7fbe499e" exitCode=0 Nov 02 23:17:53.309520 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:53.309075 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" event={"ID":"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1","Type":"ContainerDied","Data":"ea9410cc6a4527d30d9e68d51b4abef2975fd9e8243fc7b05c70951d7fbe499e"} Nov 02 23:17:53.309520 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:53.309503 5076 scope.go:122] "RemoveContainer" containerID="ea9410cc6a4527d30d9e68d51b4abef2975fd9e8243fc7b05c70951d7fbe499e" Nov 02 23:17:53.311018 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:53.310993 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:17:53.311087 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:53.311060 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:17:53.311125 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:53.311107 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:17:54.341390 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:54.341355 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" event={"ID":"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1","Type":"ContainerStarted","Data":"323da69353697cbd4bc50143494f278662d9ef246af24a61c917d0e12f21411e"} Nov 02 23:17:55.338957 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:55.332235 5076 patch_prober.go:29] interesting pod/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 23:17:55.338957 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:55.332292 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" podUID="818ebc94-d373-4c81-8c1f-d8a1a30bcbb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 23:17:55.703770 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:55.703734 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" Nov 02 23:17:56.305060 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:56.305027 5076 patch_prober.go:29] interesting pod/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 23:17:56.305231 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:56.305093 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" podUID="818ebc94-d373-4c81-8c1f-d8a1a30bcbb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 23:17:57.304656 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:57.304621 5076 patch_prober.go:29] interesting pod/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 23:17:57.305070 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:57.304693 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" podUID="818ebc94-d373-4c81-8c1f-d8a1a30bcbb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 23:17:58.303722 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:58.303683 5076 patch_prober.go:29] interesting pod/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 23:17:58.303984 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:58.303751 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" podUID="818ebc94-d373-4c81-8c1f-d8a1a30bcbb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 23:17:59.304419 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:59.304384 5076 patch_prober.go:29] interesting pod/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 23:17:59.304843 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:17:59.304451 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" podUID="818ebc94-d373-4c81-8c1f-d8a1a30bcbb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 23:18:00.308606 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:00.308578 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" Nov 02 23:18:00.309110 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:00.309089 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" Nov 02 23:18:00.309856 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:00.309835 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" Nov 02 23:18:06.655156 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:06.655122 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl"] Nov 02 23:18:06.655592 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:06.655358 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl" podUID="675b1902-02f6-49c3-bd75-1feecd543608" containerName="maas-api" containerID="cri-o://fc2785c1b757ed8b71f3d8e4c439c87859dad6d8de9a966772ae57a8be4d61e3" gracePeriod=30 Nov 02 23:18:06.859243 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:06.859213 5076 generic.go:356] "Generic (PLEG): container finished" podID="675b1902-02f6-49c3-bd75-1feecd543608" containerID="fc2785c1b757ed8b71f3d8e4c439c87859dad6d8de9a966772ae57a8be4d61e3" exitCode=0 Nov 02 23:18:06.859410 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:06.859293 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl" event={"ID":"675b1902-02f6-49c3-bd75-1feecd543608","Type":"ContainerDied","Data":"fc2785c1b757ed8b71f3d8e4c439c87859dad6d8de9a966772ae57a8be4d61e3"} Nov 02 23:18:07.188649 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:07.175432 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl" Nov 02 23:18:07.234912 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:07.234773 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/675b1902-02f6-49c3-bd75-1feecd543608-kube-api-access-vlr5h\" (UniqueName: \"kubernetes.io/projected/675b1902-02f6-49c3-bd75-1feecd543608-kube-api-access-vlr5h\") pod \"675b1902-02f6-49c3-bd75-1feecd543608\" (UID: \"675b1902-02f6-49c3-bd75-1feecd543608\") " Nov 02 23:18:07.235113 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:07.235034 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/675b1902-02f6-49c3-bd75-1feecd543608-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/675b1902-02f6-49c3-bd75-1feecd543608-openshift-service-ca\") pod \"675b1902-02f6-49c3-bd75-1feecd543608\" (UID: \"675b1902-02f6-49c3-bd75-1feecd543608\") " Nov 02 23:18:07.235176 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:07.235147 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/675b1902-02f6-49c3-bd75-1feecd543608-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/675b1902-02f6-49c3-bd75-1feecd543608-maas-api-tls\") pod \"675b1902-02f6-49c3-bd75-1feecd543608\" (UID: \"675b1902-02f6-49c3-bd75-1feecd543608\") " Nov 02 23:18:07.237435 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:07.235256 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/675b1902-02f6-49c3-bd75-1feecd543608-metrics-certs\" (UniqueName: \"kubernetes.io/secret/675b1902-02f6-49c3-bd75-1feecd543608-metrics-certs\") pod \"675b1902-02f6-49c3-bd75-1feecd543608\" (UID: \"675b1902-02f6-49c3-bd75-1feecd543608\") " Nov 02 23:18:07.241679 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:07.239148 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/675b1902-02f6-49c3-bd75-1feecd543608-openshift-service-ca" pod "675b1902-02f6-49c3-bd75-1feecd543608" (UID: "675b1902-02f6-49c3-bd75-1feecd543608"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:18:07.249224 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:07.249180 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/675b1902-02f6-49c3-bd75-1feecd543608-maas-api-tls" pod "675b1902-02f6-49c3-bd75-1feecd543608" (UID: "675b1902-02f6-49c3-bd75-1feecd543608"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:18:07.254670 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:07.254633 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/675b1902-02f6-49c3-bd75-1feecd543608-metrics-certs" pod "675b1902-02f6-49c3-bd75-1feecd543608" (UID: "675b1902-02f6-49c3-bd75-1feecd543608"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:18:07.272461 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:07.272419 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/675b1902-02f6-49c3-bd75-1feecd543608-kube-api-access-vlr5h" pod "675b1902-02f6-49c3-bd75-1feecd543608" (UID: "675b1902-02f6-49c3-bd75-1feecd543608"). InnerVolumeSpecName "kube-api-access-vlr5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:18:07.337207 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:07.337110 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlr5h\" (UniqueName: \"kubernetes.io/projected/675b1902-02f6-49c3-bd75-1feecd543608-kube-api-access-vlr5h\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:07.337207 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:07.337144 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/675b1902-02f6-49c3-bd75-1feecd543608-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:07.337207 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:07.337154 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/675b1902-02f6-49c3-bd75-1feecd543608-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:07.337207 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:07.337164 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/675b1902-02f6-49c3-bd75-1feecd543608-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:07.914924 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:07.913500 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl" event={"ID":"675b1902-02f6-49c3-bd75-1feecd543608","Type":"ContainerDied","Data":"24314cf8ece5d91bec9460eba7270338daec057d3d8e46d2ac52d22eeba837c5"} Nov 02 23:18:07.914924 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:07.913584 5076 scope.go:122] "RemoveContainer" containerID="fc2785c1b757ed8b71f3d8e4c439c87859dad6d8de9a966772ae57a8be4d61e3" Nov 02 23:18:07.914924 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:07.913761 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl" Nov 02 23:18:08.072544 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:08.072510 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl"] Nov 02 23:18:08.119924 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:08.119867 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-5qjkl"] Nov 02 23:18:09.149604 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:09.149559 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="675b1902-02f6-49c3-bd75-1feecd543608" path="/var/lib/kubelet/pods/675b1902-02f6-49c3-bd75-1feecd543608/volumes" Nov 02 23:18:09.884785 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:18:09.884737 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:18:13.208881 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:13.208849 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f46208-89d948c55-274ln"] Nov 02 23:18:13.209448 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:13.209103 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f46208-89d948c55-274ln" podUID="b1d00c89-507e-49cf-b6a5-1b03044b6626" containerName="maas-api" containerID="cri-o://9dc063970b93656cc5e0cfd700b7b781d5b07fd7a0ad23bb5f7bac8a5f4cb451" gracePeriod=30 Nov 02 23:18:13.606685 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:13.606656 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f46208-89d948c55-274ln" Nov 02 23:18:13.693921 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:13.687105 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1d00c89-507e-49cf-b6a5-1b03044b6626-kube-api-access-s6lxp\" (UniqueName: \"kubernetes.io/projected/b1d00c89-507e-49cf-b6a5-1b03044b6626-kube-api-access-s6lxp\") pod \"b1d00c89-507e-49cf-b6a5-1b03044b6626\" (UID: \"b1d00c89-507e-49cf-b6a5-1b03044b6626\") " Nov 02 23:18:13.694297 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:13.694253 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1d00c89-507e-49cf-b6a5-1b03044b6626-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1d00c89-507e-49cf-b6a5-1b03044b6626-maas-api-tls\") pod \"b1d00c89-507e-49cf-b6a5-1b03044b6626\" (UID: \"b1d00c89-507e-49cf-b6a5-1b03044b6626\") " Nov 02 23:18:13.695218 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:13.695193 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1d00c89-507e-49cf-b6a5-1b03044b6626-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1d00c89-507e-49cf-b6a5-1b03044b6626-metrics-certs\") pod \"b1d00c89-507e-49cf-b6a5-1b03044b6626\" (UID: \"b1d00c89-507e-49cf-b6a5-1b03044b6626\") " Nov 02 23:18:13.695610 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:13.695586 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1d00c89-507e-49cf-b6a5-1b03044b6626-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1d00c89-507e-49cf-b6a5-1b03044b6626-openshift-service-ca\") pod \"b1d00c89-507e-49cf-b6a5-1b03044b6626\" (UID: \"b1d00c89-507e-49cf-b6a5-1b03044b6626\") " Nov 02 23:18:13.705489 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:13.705341 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1d00c89-507e-49cf-b6a5-1b03044b6626-openshift-service-ca" pod "b1d00c89-507e-49cf-b6a5-1b03044b6626" (UID: "b1d00c89-507e-49cf-b6a5-1b03044b6626"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:18:13.706566 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:13.706538 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1d00c89-507e-49cf-b6a5-1b03044b6626-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:13.727125 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:13.721021 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1d00c89-507e-49cf-b6a5-1b03044b6626-maas-api-tls" pod "b1d00c89-507e-49cf-b6a5-1b03044b6626" (UID: "b1d00c89-507e-49cf-b6a5-1b03044b6626"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:18:13.727125 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:13.721764 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1d00c89-507e-49cf-b6a5-1b03044b6626-metrics-certs" pod "b1d00c89-507e-49cf-b6a5-1b03044b6626" (UID: "b1d00c89-507e-49cf-b6a5-1b03044b6626"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:18:13.792205 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:13.792095 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1d00c89-507e-49cf-b6a5-1b03044b6626-kube-api-access-s6lxp" pod "b1d00c89-507e-49cf-b6a5-1b03044b6626" (UID: "b1d00c89-507e-49cf-b6a5-1b03044b6626"). InnerVolumeSpecName "kube-api-access-s6lxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:18:13.808293 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:13.808216 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1d00c89-507e-49cf-b6a5-1b03044b6626-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:13.808293 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:13.808251 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1d00c89-507e-49cf-b6a5-1b03044b6626-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:13.808293 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:13.808266 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6lxp\" (UniqueName: \"kubernetes.io/projected/b1d00c89-507e-49cf-b6a5-1b03044b6626-kube-api-access-s6lxp\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:13.981968 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:13.981927 5076 generic.go:356] "Generic (PLEG): container finished" podID="b1d00c89-507e-49cf-b6a5-1b03044b6626" containerID="9dc063970b93656cc5e0cfd700b7b781d5b07fd7a0ad23bb5f7bac8a5f4cb451" exitCode=0 Nov 02 23:18:13.982166 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:13.981985 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f46208-89d948c55-274ln" Nov 02 23:18:13.982166 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:13.982016 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f46208-89d948c55-274ln" event={"ID":"b1d00c89-507e-49cf-b6a5-1b03044b6626","Type":"ContainerDied","Data":"9dc063970b93656cc5e0cfd700b7b781d5b07fd7a0ad23bb5f7bac8a5f4cb451"} Nov 02 23:18:13.982166 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:13.982084 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f46208-89d948c55-274ln" event={"ID":"b1d00c89-507e-49cf-b6a5-1b03044b6626","Type":"ContainerDied","Data":"2a7866ea8721213a53517553ac8485e288038fadea4ea590d1f999ad2f9fa5b9"} Nov 02 23:18:13.982166 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:13.982115 5076 scope.go:122] "RemoveContainer" containerID="9dc063970b93656cc5e0cfd700b7b781d5b07fd7a0ad23bb5f7bac8a5f4cb451" Nov 02 23:18:13.993201 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:13.993183 5076 scope.go:122] "RemoveContainer" containerID="9dc063970b93656cc5e0cfd700b7b781d5b07fd7a0ad23bb5f7bac8a5f4cb451" Nov 02 23:18:13.993474 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:18:13.993456 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dc063970b93656cc5e0cfd700b7b781d5b07fd7a0ad23bb5f7bac8a5f4cb451\": container with ID starting with 9dc063970b93656cc5e0cfd700b7b781d5b07fd7a0ad23bb5f7bac8a5f4cb451 not found: ID does not exist" containerID="9dc063970b93656cc5e0cfd700b7b781d5b07fd7a0ad23bb5f7bac8a5f4cb451" Nov 02 23:18:13.993540 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:13.993482 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dc063970b93656cc5e0cfd700b7b781d5b07fd7a0ad23bb5f7bac8a5f4cb451"} err="failed to get container status \"9dc063970b93656cc5e0cfd700b7b781d5b07fd7a0ad23bb5f7bac8a5f4cb451\": rpc error: code = NotFound desc = could not find container \"9dc063970b93656cc5e0cfd700b7b781d5b07fd7a0ad23bb5f7bac8a5f4cb451\": container with ID starting with 9dc063970b93656cc5e0cfd700b7b781d5b07fd7a0ad23bb5f7bac8a5f4cb451 not found: ID does not exist" Nov 02 23:18:14.139181 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:14.139083 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f46208-89d948c55-274ln"] Nov 02 23:18:14.186218 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:14.186184 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f46208-89d948c55-274ln"] Nov 02 23:18:14.991843 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:14.991807 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-68e067-7b49597786-b7bwj"] Nov 02 23:18:14.992562 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:14.992544 5076 cpu_manager.go:414] "containerMap: removing container" podUID="675b1902-02f6-49c3-bd75-1feecd543608" containerName="maas-api" Nov 02 23:18:14.992624 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:14.992564 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="675b1902-02f6-49c3-bd75-1feecd543608" containerName="maas-api" Nov 02 23:18:14.992624 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:14.992594 5076 cpu_manager.go:414] "containerMap: removing container" podUID="b1d00c89-507e-49cf-b6a5-1b03044b6626" containerName="maas-api" Nov 02 23:18:14.992624 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:14.992599 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1d00c89-507e-49cf-b6a5-1b03044b6626" containerName="maas-api" Nov 02 23:18:14.992726 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:14.992685 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1d00c89-507e-49cf-b6a5-1b03044b6626" containerName="maas-api" Nov 02 23:18:14.992726 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:14.992700 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="675b1902-02f6-49c3-bd75-1feecd543608" containerName="maas-api" Nov 02 23:18:14.996060 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:14.996040 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68e067-7b49597786-b7bwj" Nov 02 23:18:15.049119 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:15.049089 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-68e067-7b49597786-b7bwj"] Nov 02 23:18:15.092448 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:15.092412 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-68e067-serving-cert\"" Nov 02 23:18:15.141456 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:15.141422 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cd608db-c160-4c05-b8d5-aa3cf80288b7-metrics-certs\") pod \"maas-api-e2e-api-a-68e067-7b49597786-b7bwj\" (UID: \"7cd608db-c160-4c05-b8d5-aa3cf80288b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68e067-7b49597786-b7bwj" Nov 02 23:18:15.141764 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:15.141733 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cd608db-c160-4c05-b8d5-aa3cf80288b7-maas-api-tls\") pod \"maas-api-e2e-api-a-68e067-7b49597786-b7bwj\" (UID: \"7cd608db-c160-4c05-b8d5-aa3cf80288b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68e067-7b49597786-b7bwj" Nov 02 23:18:15.142269 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:15.142198 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cd608db-c160-4c05-b8d5-aa3cf80288b7-openshift-service-ca\") pod \"maas-api-e2e-api-a-68e067-7b49597786-b7bwj\" (UID: \"7cd608db-c160-4c05-b8d5-aa3cf80288b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68e067-7b49597786-b7bwj" Nov 02 23:18:15.142531 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:15.142505 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8tkq\" (UniqueName: \"kubernetes.io/projected/7cd608db-c160-4c05-b8d5-aa3cf80288b7-kube-api-access-l8tkq\") pod \"maas-api-e2e-api-a-68e067-7b49597786-b7bwj\" (UID: \"7cd608db-c160-4c05-b8d5-aa3cf80288b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68e067-7b49597786-b7bwj" Nov 02 23:18:15.163493 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:15.163452 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1d00c89-507e-49cf-b6a5-1b03044b6626" path="/var/lib/kubelet/pods/b1d00c89-507e-49cf-b6a5-1b03044b6626/volumes" Nov 02 23:18:15.204498 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:15.204467 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7"] Nov 02 23:18:15.204728 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:15.204702 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" podUID="3751da0e-99b6-4f2c-836e-699267f1428d" containerName="istio-proxy" containerID="cri-o://9f9e7a539b3f290115ff21f86c5bfe2ade555ff64c7699eee91e34c4b032447c" gracePeriod=30 Nov 02 23:18:15.208701 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:15.208670 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx"] Nov 02 23:18:15.208956 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:15.208927 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" podUID="d18c1e9a-d8c2-4438-8e96-2393552f495e" containerName="istio-proxy" containerID="cri-o://9e43b7e56f165e0d0165337b511036ea62356c7a748236f22035f656db077dc2" gracePeriod=30 Nov 02 23:18:15.213609 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:15.213584 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8"] Nov 02 23:18:15.213877 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:15.213808 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" podUID="73f3edf1-31bb-41a5-abb8-8a97ab98b1b4" containerName="istio-proxy" containerID="cri-o://dba449aa9a5e3ef461583ed7bdd9e17a1ee3f32938ae213fb4a53998b535a5fa" gracePeriod=30 Nov 02 23:18:15.218832 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:15.218807 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls"] Nov 02 23:18:15.219049 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:15.219024 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" podUID="0c2d7a91-9af2-40b7-a010-1a19260cf6f9" containerName="istio-proxy" containerID="cri-o://12ea9b1a63208befc24d12c9fa7be2120eaefb2fcaece2475a13f4a73265eed7" gracePeriod=30 Nov 02 23:18:15.244457 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:15.244367 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cd608db-c160-4c05-b8d5-aa3cf80288b7-maas-api-tls\") pod \"maas-api-e2e-api-a-68e067-7b49597786-b7bwj\" (UID: \"7cd608db-c160-4c05-b8d5-aa3cf80288b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68e067-7b49597786-b7bwj" Nov 02 23:18:15.244625 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:15.244503 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cd608db-c160-4c05-b8d5-aa3cf80288b7-openshift-service-ca\") pod \"maas-api-e2e-api-a-68e067-7b49597786-b7bwj\" (UID: \"7cd608db-c160-4c05-b8d5-aa3cf80288b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68e067-7b49597786-b7bwj" Nov 02 23:18:15.244625 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:15.244566 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8tkq\" (UniqueName: \"kubernetes.io/projected/7cd608db-c160-4c05-b8d5-aa3cf80288b7-kube-api-access-l8tkq\") pod \"maas-api-e2e-api-a-68e067-7b49597786-b7bwj\" (UID: \"7cd608db-c160-4c05-b8d5-aa3cf80288b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68e067-7b49597786-b7bwj" Nov 02 23:18:15.244766 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:15.244647 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cd608db-c160-4c05-b8d5-aa3cf80288b7-metrics-certs\") pod \"maas-api-e2e-api-a-68e067-7b49597786-b7bwj\" (UID: \"7cd608db-c160-4c05-b8d5-aa3cf80288b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68e067-7b49597786-b7bwj" Nov 02 23:18:15.244879 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:18:15.244853 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-68e067-serving-cert: secret "maas-api-e2e-api-a-68e067-serving-cert" not found Nov 02 23:18:15.244972 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:18:15.244940 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cd608db-c160-4c05-b8d5-aa3cf80288b7-maas-api-tls podName:7cd608db-c160-4c05-b8d5-aa3cf80288b7 nodeName:}" failed. No retries permitted until 2026-11-02 23:18:15.744918725 +0000 UTC m=+1544.713450505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7cd608db-c160-4c05-b8d5-aa3cf80288b7-maas-api-tls") pod "maas-api-e2e-api-a-68e067-7b49597786-b7bwj" (UID: "7cd608db-c160-4c05-b8d5-aa3cf80288b7") : secret "maas-api-e2e-api-a-68e067-serving-cert" not found Nov 02 23:18:15.245483 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:15.245457 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cd608db-c160-4c05-b8d5-aa3cf80288b7-openshift-service-ca\") pod \"maas-api-e2e-api-a-68e067-7b49597786-b7bwj\" (UID: \"7cd608db-c160-4c05-b8d5-aa3cf80288b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68e067-7b49597786-b7bwj" Nov 02 23:18:15.247657 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:15.247630 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cd608db-c160-4c05-b8d5-aa3cf80288b7-metrics-certs\") pod \"maas-api-e2e-api-a-68e067-7b49597786-b7bwj\" (UID: \"7cd608db-c160-4c05-b8d5-aa3cf80288b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68e067-7b49597786-b7bwj" Nov 02 23:18:15.417958 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:15.416437 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8tkq\" (UniqueName: \"kubernetes.io/projected/7cd608db-c160-4c05-b8d5-aa3cf80288b7-kube-api-access-l8tkq\") pod \"maas-api-e2e-api-a-68e067-7b49597786-b7bwj\" (UID: \"7cd608db-c160-4c05-b8d5-aa3cf80288b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68e067-7b49597786-b7bwj" Nov 02 23:18:15.773416 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:15.773356 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cd608db-c160-4c05-b8d5-aa3cf80288b7-maas-api-tls\") pod \"maas-api-e2e-api-a-68e067-7b49597786-b7bwj\" (UID: \"7cd608db-c160-4c05-b8d5-aa3cf80288b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68e067-7b49597786-b7bwj" Nov 02 23:18:15.775832 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:15.775801 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cd608db-c160-4c05-b8d5-aa3cf80288b7-maas-api-tls\") pod \"maas-api-e2e-api-a-68e067-7b49597786-b7bwj\" (UID: \"7cd608db-c160-4c05-b8d5-aa3cf80288b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68e067-7b49597786-b7bwj" Nov 02 23:18:15.908867 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:15.908833 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68e067-7b49597786-b7bwj" Nov 02 23:18:16.275549 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:16.275460 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-68e067-7b49597786-b7bwj"] Nov 02 23:18:16.276038 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:18:16.275885 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd608db_c160_4c05_b8d5_aa3cf80288b7.slice/crio-3038436ced2ca0642daf2599a8700c43460d2dc84475306c845e8af8ff2a0e94 WatchSource:0}: Error finding container 3038436ced2ca0642daf2599a8700c43460d2dc84475306c845e8af8ff2a0e94: Status 404 returned error can't find the container with id 3038436ced2ca0642daf2599a8700c43460d2dc84475306c845e8af8ff2a0e94 Nov 02 23:18:17.006289 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:17.006238 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68e067-7b49597786-b7bwj" event={"ID":"7cd608db-c160-4c05-b8d5-aa3cf80288b7","Type":"ContainerStarted","Data":"3038436ced2ca0642daf2599a8700c43460d2dc84475306c845e8af8ff2a0e94"} Nov 02 23:18:18.018557 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:18.018397 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68e067-7b49597786-b7bwj" event={"ID":"7cd608db-c160-4c05-b8d5-aa3cf80288b7","Type":"ContainerStarted","Data":"d24ab1384a2ef77a6454dd2924beecb51bcf7e1c5e76743d3c2e00197940b135"} Nov 02 23:18:18.019058 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:18.018612 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68e067-7b49597786-b7bwj" Nov 02 23:18:18.130131 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:18.130077 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68e067-7b49597786-b7bwj" podStartSLOduration=3.055207816 podStartE2EDuration="4.130062551s" totalImagesPullingTime="1.074854735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:18:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:18:18.129076599 +0000 UTC m=+1547.097608383" watchObservedRunningTime="2026-11-02 23:18:18.130062551 +0000 UTC m=+1547.098594335" Nov 02 23:18:19.309324 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:19.309286 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:18:19.309723 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:19.309338 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" podUID="0c2d7a91-9af2-40b7-a010-1a19260cf6f9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:18:20.120922 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:19.931882 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5"] Nov 02 23:18:20.120922 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.119129 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" podUID="f19211b1-29a9-4e75-bd8a-ab654254a5a1" containerName="istio-proxy" containerID="cri-o://6e7d7ee86caa0d0a1f695f502d590ab2f7b8913972325d81589d247a5d059ad5" gracePeriod=30 Nov 02 23:18:20.407624 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.405642 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 02 23:18:20.407624 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.405699 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" podUID="3751da0e-99b6-4f2c-836e-699267f1428d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 02 23:18:20.535811 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:18:20.535740 5076 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3751da0e_99b6_4f2c_836e_699267f1428d.slice/crio-9f9e7a539b3f290115ff21f86c5bfe2ade555ff64c7699eee91e34c4b032447c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c2d7a91_9af2_40b7_a010_1a19260cf6f9.slice/crio-conmon-12ea9b1a63208befc24d12c9fa7be2120eaefb2fcaece2475a13f4a73265eed7.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73f3edf1_31bb_41a5_abb8_8a97ab98b1b4.slice/crio-dba449aa9a5e3ef461583ed7bdd9e17a1ee3f32938ae213fb4a53998b535a5fa.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd18c1e9a_d8c2_4438_8e96_2393552f495e.slice/crio-9e43b7e56f165e0d0165337b511036ea62356c7a748236f22035f656db077dc2.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c2d7a91_9af2_40b7_a010_1a19260cf6f9.slice/crio-12ea9b1a63208befc24d12c9fa7be2120eaefb2fcaece2475a13f4a73265eed7.scope\": RecentStats: unable to find data in memory cache]" Nov 02 23:18:20.651533 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.651497 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" Nov 02 23:18:20.758530 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.758445 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" Nov 02 23:18:20.791910 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.789763 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-istio-data\") pod \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\" (UID: \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\") " Nov 02 23:18:20.791910 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.790164 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-workload-socket\") pod \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\" (UID: \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\") " Nov 02 23:18:20.791910 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.790441 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-credential-socket\") pod \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\" (UID: \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\") " Nov 02 23:18:20.791910 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.790725 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-istio-podinfo\") pod \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\" (UID: \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\") " Nov 02 23:18:20.791910 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.790774 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-istio-envoy\") pod \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\" (UID: \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\") " Nov 02 23:18:20.791910 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.790098 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-istio-data" pod "73f3edf1-31bb-41a5-abb8-8a97ab98b1b4" (UID: "73f3edf1-31bb-41a5-abb8-8a97ab98b1b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:20.791910 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.790377 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-workload-socket" pod "73f3edf1-31bb-41a5-abb8-8a97ab98b1b4" (UID: "73f3edf1-31bb-41a5-abb8-8a97ab98b1b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:20.791910 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.790673 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-credential-socket" pod "73f3edf1-31bb-41a5-abb8-8a97ab98b1b4" (UID: "73f3edf1-31bb-41a5-abb8-8a97ab98b1b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:20.791910 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.791758 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-workload-certs\") pod \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\" (UID: \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\") " Nov 02 23:18:20.791910 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.791798 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-istiod-ca-cert\") pod \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\" (UID: \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\") " Nov 02 23:18:20.791910 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.791830 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-kube-api-access-wmdpq\" (UniqueName: \"kubernetes.io/projected/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-kube-api-access-wmdpq\") pod \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\" (UID: \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\") " Nov 02 23:18:20.793052 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.792065 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-istio-token\" (UniqueName: \"kubernetes.io/projected/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-istio-token\") pod \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\" (UID: \"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4\") " Nov 02 23:18:20.793052 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.792915 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:20.793052 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.792939 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:20.793052 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.792953 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:20.793052 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.792914 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-workload-certs" pod "73f3edf1-31bb-41a5-abb8-8a97ab98b1b4" (UID: "73f3edf1-31bb-41a5-abb8-8a97ab98b1b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:20.795919 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.795217 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-istiod-ca-cert" pod "73f3edf1-31bb-41a5-abb8-8a97ab98b1b4" (UID: "73f3edf1-31bb-41a5-abb8-8a97ab98b1b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:18:20.800580 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.800536 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-istio-token" pod "73f3edf1-31bb-41a5-abb8-8a97ab98b1b4" (UID: "73f3edf1-31bb-41a5-abb8-8a97ab98b1b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:18:20.800830 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.800798 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-istio-envoy" pod "73f3edf1-31bb-41a5-abb8-8a97ab98b1b4" (UID: "73f3edf1-31bb-41a5-abb8-8a97ab98b1b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:20.803493 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.803467 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-kube-api-access-wmdpq" pod "73f3edf1-31bb-41a5-abb8-8a97ab98b1b4" (UID: "73f3edf1-31bb-41a5-abb8-8a97ab98b1b4"). InnerVolumeSpecName "kube-api-access-wmdpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:18:20.806978 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.806928 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-istio-podinfo" pod "73f3edf1-31bb-41a5-abb8-8a97ab98b1b4" (UID: "73f3edf1-31bb-41a5-abb8-8a97ab98b1b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:18:20.878142 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.877937 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" Nov 02 23:18:20.882330 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.882226 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" Nov 02 23:18:20.893973 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.893923 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3751da0e-99b6-4f2c-836e-699267f1428d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3751da0e-99b6-4f2c-836e-699267f1428d-istiod-ca-cert\") pod \"3751da0e-99b6-4f2c-836e-699267f1428d\" (UID: \"3751da0e-99b6-4f2c-836e-699267f1428d\") " Nov 02 23:18:20.894385 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.894334 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3751da0e-99b6-4f2c-836e-699267f1428d-kube-api-access-cvvsl\" (UniqueName: \"kubernetes.io/projected/3751da0e-99b6-4f2c-836e-699267f1428d-kube-api-access-cvvsl\") pod \"3751da0e-99b6-4f2c-836e-699267f1428d\" (UID: \"3751da0e-99b6-4f2c-836e-699267f1428d\") " Nov 02 23:18:20.896060 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.894582 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3751da0e-99b6-4f2c-836e-699267f1428d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3751da0e-99b6-4f2c-836e-699267f1428d-istio-podinfo\") pod \"3751da0e-99b6-4f2c-836e-699267f1428d\" (UID: \"3751da0e-99b6-4f2c-836e-699267f1428d\") " Nov 02 23:18:20.896060 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.894659 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3751da0e-99b6-4f2c-836e-699267f1428d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3751da0e-99b6-4f2c-836e-699267f1428d-credential-socket\") pod \"3751da0e-99b6-4f2c-836e-699267f1428d\" (UID: \"3751da0e-99b6-4f2c-836e-699267f1428d\") " Nov 02 23:18:20.896060 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.894798 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3751da0e-99b6-4f2c-836e-699267f1428d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3751da0e-99b6-4f2c-836e-699267f1428d-istio-envoy\") pod \"3751da0e-99b6-4f2c-836e-699267f1428d\" (UID: \"3751da0e-99b6-4f2c-836e-699267f1428d\") " Nov 02 23:18:20.896060 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.894955 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3751da0e-99b6-4f2c-836e-699267f1428d-istio-token\" (UniqueName: \"kubernetes.io/projected/3751da0e-99b6-4f2c-836e-699267f1428d-istio-token\") pod \"3751da0e-99b6-4f2c-836e-699267f1428d\" (UID: \"3751da0e-99b6-4f2c-836e-699267f1428d\") " Nov 02 23:18:20.896060 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.895003 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3751da0e-99b6-4f2c-836e-699267f1428d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3751da0e-99b6-4f2c-836e-699267f1428d-workload-certs\") pod \"3751da0e-99b6-4f2c-836e-699267f1428d\" (UID: \"3751da0e-99b6-4f2c-836e-699267f1428d\") " Nov 02 23:18:20.896060 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.895046 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3751da0e-99b6-4f2c-836e-699267f1428d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3751da0e-99b6-4f2c-836e-699267f1428d-workload-socket\") pod \"3751da0e-99b6-4f2c-836e-699267f1428d\" (UID: \"3751da0e-99b6-4f2c-836e-699267f1428d\") " Nov 02 23:18:20.896060 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.895130 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3751da0e-99b6-4f2c-836e-699267f1428d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3751da0e-99b6-4f2c-836e-699267f1428d-istio-data\") pod \"3751da0e-99b6-4f2c-836e-699267f1428d\" (UID: \"3751da0e-99b6-4f2c-836e-699267f1428d\") " Nov 02 23:18:20.896060 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.895866 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:20.896060 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.895907 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:20.896060 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.895923 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:20.896060 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.895937 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:20.896060 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.895951 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:20.896060 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.895965 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmdpq\" (UniqueName: \"kubernetes.io/projected/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4-kube-api-access-wmdpq\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:20.898963 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.898531 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3751da0e-99b6-4f2c-836e-699267f1428d-istiod-ca-cert" pod "3751da0e-99b6-4f2c-836e-699267f1428d" (UID: "3751da0e-99b6-4f2c-836e-699267f1428d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:18:20.901444 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.900185 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3751da0e-99b6-4f2c-836e-699267f1428d-workload-certs" pod "3751da0e-99b6-4f2c-836e-699267f1428d" (UID: "3751da0e-99b6-4f2c-836e-699267f1428d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:20.901444 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.901133 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3751da0e-99b6-4f2c-836e-699267f1428d-workload-socket" pod "3751da0e-99b6-4f2c-836e-699267f1428d" (UID: "3751da0e-99b6-4f2c-836e-699267f1428d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:20.901444 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.901298 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3751da0e-99b6-4f2c-836e-699267f1428d-istio-data" pod "3751da0e-99b6-4f2c-836e-699267f1428d" (UID: "3751da0e-99b6-4f2c-836e-699267f1428d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:20.901771 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.901736 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3751da0e-99b6-4f2c-836e-699267f1428d-credential-socket" pod "3751da0e-99b6-4f2c-836e-699267f1428d" (UID: "3751da0e-99b6-4f2c-836e-699267f1428d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:20.909049 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.908996 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3751da0e-99b6-4f2c-836e-699267f1428d-istio-envoy" pod "3751da0e-99b6-4f2c-836e-699267f1428d" (UID: "3751da0e-99b6-4f2c-836e-699267f1428d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:20.909049 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.909024 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3751da0e-99b6-4f2c-836e-699267f1428d-kube-api-access-cvvsl" pod "3751da0e-99b6-4f2c-836e-699267f1428d" (UID: "3751da0e-99b6-4f2c-836e-699267f1428d"). InnerVolumeSpecName "kube-api-access-cvvsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:18:20.912520 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.911416 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3751da0e-99b6-4f2c-836e-699267f1428d-istio-token" pod "3751da0e-99b6-4f2c-836e-699267f1428d" (UID: "3751da0e-99b6-4f2c-836e-699267f1428d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:18:20.912520 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.911506 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3751da0e-99b6-4f2c-836e-699267f1428d-istio-podinfo" pod "3751da0e-99b6-4f2c-836e-699267f1428d" (UID: "3751da0e-99b6-4f2c-836e-699267f1428d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:18:20.996773 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.996737 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-istiod-ca-cert\") pod \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\" (UID: \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\") " Nov 02 23:18:20.996976 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.996824 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d18c1e9a-d8c2-4438-8e96-2393552f495e-istio-token\" (UniqueName: \"kubernetes.io/projected/d18c1e9a-d8c2-4438-8e96-2393552f495e-istio-token\") pod \"d18c1e9a-d8c2-4438-8e96-2393552f495e\" (UID: \"d18c1e9a-d8c2-4438-8e96-2393552f495e\") " Nov 02 23:18:20.996976 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.996849 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-kube-api-access-xbh8l\" (UniqueName: \"kubernetes.io/projected/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-kube-api-access-xbh8l\") pod \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\" (UID: \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\") " Nov 02 23:18:20.996976 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.996912 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d18c1e9a-d8c2-4438-8e96-2393552f495e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18c1e9a-d8c2-4438-8e96-2393552f495e-credential-socket\") pod \"d18c1e9a-d8c2-4438-8e96-2393552f495e\" (UID: \"d18c1e9a-d8c2-4438-8e96-2393552f495e\") " Nov 02 23:18:20.996976 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.996931 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d18c1e9a-d8c2-4438-8e96-2393552f495e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d18c1e9a-d8c2-4438-8e96-2393552f495e-istio-envoy\") pod \"d18c1e9a-d8c2-4438-8e96-2393552f495e\" (UID: \"d18c1e9a-d8c2-4438-8e96-2393552f495e\") " Nov 02 23:18:20.996976 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.996966 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-workload-socket\") pod \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\" (UID: \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\") " Nov 02 23:18:20.997299 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.997003 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-workload-certs\") pod \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\" (UID: \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\") " Nov 02 23:18:20.997299 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.997102 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d18c1e9a-d8c2-4438-8e96-2393552f495e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d18c1e9a-d8c2-4438-8e96-2393552f495e-istiod-ca-cert\") pod \"d18c1e9a-d8c2-4438-8e96-2393552f495e\" (UID: \"d18c1e9a-d8c2-4438-8e96-2393552f495e\") " Nov 02 23:18:20.997299 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.997155 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d18c1e9a-d8c2-4438-8e96-2393552f495e-kube-api-access-55xrf\" (UniqueName: \"kubernetes.io/projected/d18c1e9a-d8c2-4438-8e96-2393552f495e-kube-api-access-55xrf\") pod \"d18c1e9a-d8c2-4438-8e96-2393552f495e\" (UID: \"d18c1e9a-d8c2-4438-8e96-2393552f495e\") " Nov 02 23:18:20.997299 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.997189 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d18c1e9a-d8c2-4438-8e96-2393552f495e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d18c1e9a-d8c2-4438-8e96-2393552f495e-istio-data\") pod \"d18c1e9a-d8c2-4438-8e96-2393552f495e\" (UID: \"d18c1e9a-d8c2-4438-8e96-2393552f495e\") " Nov 02 23:18:20.997299 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.997236 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-istiod-ca-cert" pod "0c2d7a91-9af2-40b7-a010-1a19260cf6f9" (UID: "0c2d7a91-9af2-40b7-a010-1a19260cf6f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:18:20.997299 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.997252 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-istio-envoy\") pod \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\" (UID: \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\") " Nov 02 23:18:20.997613 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.997320 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-istio-token\" (UniqueName: \"kubernetes.io/projected/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-istio-token\") pod \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\" (UID: \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\") " Nov 02 23:18:20.997613 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.997335 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d18c1e9a-d8c2-4438-8e96-2393552f495e-credential-socket" pod "d18c1e9a-d8c2-4438-8e96-2393552f495e" (UID: "d18c1e9a-d8c2-4438-8e96-2393552f495e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:20.997613 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.997354 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d18c1e9a-d8c2-4438-8e96-2393552f495e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18c1e9a-d8c2-4438-8e96-2393552f495e-workload-socket\") pod \"d18c1e9a-d8c2-4438-8e96-2393552f495e\" (UID: \"d18c1e9a-d8c2-4438-8e96-2393552f495e\") " Nov 02 23:18:20.997613 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.997465 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-istio-podinfo\") pod \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\" (UID: \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\") " Nov 02 23:18:20.997613 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.997518 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d18c1e9a-d8c2-4438-8e96-2393552f495e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d18c1e9a-d8c2-4438-8e96-2393552f495e-istio-podinfo\") pod \"d18c1e9a-d8c2-4438-8e96-2393552f495e\" (UID: \"d18c1e9a-d8c2-4438-8e96-2393552f495e\") " Nov 02 23:18:20.997613 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.997568 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-istio-data\") pod \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\" (UID: \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\") " Nov 02 23:18:20.997613 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.997611 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d18c1e9a-d8c2-4438-8e96-2393552f495e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d18c1e9a-d8c2-4438-8e96-2393552f495e-workload-certs\") pod \"d18c1e9a-d8c2-4438-8e96-2393552f495e\" (UID: \"d18c1e9a-d8c2-4438-8e96-2393552f495e\") " Nov 02 23:18:20.997992 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.997665 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-credential-socket\") pod \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\" (UID: \"0c2d7a91-9af2-40b7-a010-1a19260cf6f9\") " Nov 02 23:18:20.999198 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.998415 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d18c1e9a-d8c2-4438-8e96-2393552f495e-istiod-ca-cert" pod "d18c1e9a-d8c2-4438-8e96-2393552f495e" (UID: "d18c1e9a-d8c2-4438-8e96-2393552f495e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:18:20.999198 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.998709 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d18c1e9a-d8c2-4438-8e96-2393552f495e-istio-data" pod "d18c1e9a-d8c2-4438-8e96-2393552f495e" (UID: "d18c1e9a-d8c2-4438-8e96-2393552f495e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:20.999198 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.998788 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d18c1e9a-d8c2-4438-8e96-2393552f495e-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:20.999198 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.998811 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3751da0e-99b6-4f2c-836e-699267f1428d-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:20.999198 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.998826 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3751da0e-99b6-4f2c-836e-699267f1428d-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:20.999198 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.998842 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3751da0e-99b6-4f2c-836e-699267f1428d-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:20.999198 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.998857 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3751da0e-99b6-4f2c-836e-699267f1428d-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:20.999198 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.998871 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3751da0e-99b6-4f2c-836e-699267f1428d-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:20.999198 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.998885 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvvsl\" (UniqueName: \"kubernetes.io/projected/3751da0e-99b6-4f2c-836e-699267f1428d-kube-api-access-cvvsl\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:20.999198 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.998920 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:20.999198 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.998935 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3751da0e-99b6-4f2c-836e-699267f1428d-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:20.999198 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.998950 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3751da0e-99b6-4f2c-836e-699267f1428d-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:20.999198 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.998959 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-workload-socket" pod "0c2d7a91-9af2-40b7-a010-1a19260cf6f9" (UID: "0c2d7a91-9af2-40b7-a010-1a19260cf6f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:20.999198 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.998965 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18c1e9a-d8c2-4438-8e96-2393552f495e-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:20.999198 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.999001 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3751da0e-99b6-4f2c-836e-699267f1428d-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:20.999198 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.999044 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-istio-data" pod "0c2d7a91-9af2-40b7-a010-1a19260cf6f9" (UID: "0c2d7a91-9af2-40b7-a010-1a19260cf6f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:20.999198 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.999165 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-workload-certs" pod "0c2d7a91-9af2-40b7-a010-1a19260cf6f9" (UID: "0c2d7a91-9af2-40b7-a010-1a19260cf6f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:21.000243 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.999313 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-credential-socket" pod "0c2d7a91-9af2-40b7-a010-1a19260cf6f9" (UID: "0c2d7a91-9af2-40b7-a010-1a19260cf6f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:21.000243 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.999554 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d18c1e9a-d8c2-4438-8e96-2393552f495e-workload-certs" pod "d18c1e9a-d8c2-4438-8e96-2393552f495e" (UID: "d18c1e9a-d8c2-4438-8e96-2393552f495e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:21.000243 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:20.999989 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d18c1e9a-d8c2-4438-8e96-2393552f495e-workload-socket" pod "d18c1e9a-d8c2-4438-8e96-2393552f495e" (UID: "d18c1e9a-d8c2-4438-8e96-2393552f495e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:21.000581 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.000543 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-kube-api-access-xbh8l" pod "0c2d7a91-9af2-40b7-a010-1a19260cf6f9" (UID: "0c2d7a91-9af2-40b7-a010-1a19260cf6f9"). InnerVolumeSpecName "kube-api-access-xbh8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:18:21.000697 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.000609 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-istio-token" pod "0c2d7a91-9af2-40b7-a010-1a19260cf6f9" (UID: "0c2d7a91-9af2-40b7-a010-1a19260cf6f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:18:21.000697 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.000639 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-istio-envoy" pod "0c2d7a91-9af2-40b7-a010-1a19260cf6f9" (UID: "0c2d7a91-9af2-40b7-a010-1a19260cf6f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:21.000697 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.000650 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d18c1e9a-d8c2-4438-8e96-2393552f495e-istio-token" pod "d18c1e9a-d8c2-4438-8e96-2393552f495e" (UID: "d18c1e9a-d8c2-4438-8e96-2393552f495e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:18:21.000932 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.000914 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-istio-podinfo" pod "0c2d7a91-9af2-40b7-a010-1a19260cf6f9" (UID: "0c2d7a91-9af2-40b7-a010-1a19260cf6f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:18:21.001108 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.001083 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d18c1e9a-d8c2-4438-8e96-2393552f495e-istio-envoy" pod "d18c1e9a-d8c2-4438-8e96-2393552f495e" (UID: "d18c1e9a-d8c2-4438-8e96-2393552f495e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:21.001195 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.001108 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d18c1e9a-d8c2-4438-8e96-2393552f495e-kube-api-access-55xrf" pod "d18c1e9a-d8c2-4438-8e96-2393552f495e" (UID: "d18c1e9a-d8c2-4438-8e96-2393552f495e"). InnerVolumeSpecName "kube-api-access-55xrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:18:21.001497 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.001478 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d18c1e9a-d8c2-4438-8e96-2393552f495e-istio-podinfo" pod "d18c1e9a-d8c2-4438-8e96-2393552f495e" (UID: "d18c1e9a-d8c2-4438-8e96-2393552f495e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:18:21.098872 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.098837 5076 generic.go:356] "Generic (PLEG): container finished" podID="0c2d7a91-9af2-40b7-a010-1a19260cf6f9" containerID="12ea9b1a63208befc24d12c9fa7be2120eaefb2fcaece2475a13f4a73265eed7" exitCode=0 Nov 02 23:18:21.099077 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.098969 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" event={"ID":"0c2d7a91-9af2-40b7-a010-1a19260cf6f9","Type":"ContainerDied","Data":"12ea9b1a63208befc24d12c9fa7be2120eaefb2fcaece2475a13f4a73265eed7"} Nov 02 23:18:21.099077 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.099008 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" Nov 02 23:18:21.099077 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.099025 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls" event={"ID":"0c2d7a91-9af2-40b7-a010-1a19260cf6f9","Type":"ContainerDied","Data":"8ddf5012b64f1baa714f20ef97822bb926a78fb2da35a08c9fc222fe277a305e"} Nov 02 23:18:21.099077 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.099051 5076 scope.go:122] "RemoveContainer" containerID="12ea9b1a63208befc24d12c9fa7be2120eaefb2fcaece2475a13f4a73265eed7" Nov 02 23:18:21.100148 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.100075 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55xrf\" (UniqueName: \"kubernetes.io/projected/d18c1e9a-d8c2-4438-8e96-2393552f495e-kube-api-access-55xrf\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:21.100148 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.100099 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d18c1e9a-d8c2-4438-8e96-2393552f495e-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:21.100148 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.100114 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:21.100148 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.100127 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:21.100148 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.100140 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18c1e9a-d8c2-4438-8e96-2393552f495e-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:21.100449 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.100153 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:21.100449 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.100168 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d18c1e9a-d8c2-4438-8e96-2393552f495e-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:21.100449 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.100182 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:21.100449 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.100196 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d18c1e9a-d8c2-4438-8e96-2393552f495e-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:21.100449 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.100210 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:21.100449 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.100223 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d18c1e9a-d8c2-4438-8e96-2393552f495e-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:21.100449 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.100236 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbh8l\" (UniqueName: \"kubernetes.io/projected/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-kube-api-access-xbh8l\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:21.100449 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.100249 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d18c1e9a-d8c2-4438-8e96-2393552f495e-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:21.100449 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.100262 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:21.100449 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.100276 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c2d7a91-9af2-40b7-a010-1a19260cf6f9-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:21.100975 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.100571 5076 generic.go:356] "Generic (PLEG): container finished" podID="3751da0e-99b6-4f2c-836e-699267f1428d" containerID="9f9e7a539b3f290115ff21f86c5bfe2ade555ff64c7699eee91e34c4b032447c" exitCode=0 Nov 02 23:18:21.100975 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.100746 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" event={"ID":"3751da0e-99b6-4f2c-836e-699267f1428d","Type":"ContainerDied","Data":"9f9e7a539b3f290115ff21f86c5bfe2ade555ff64c7699eee91e34c4b032447c"} Nov 02 23:18:21.100975 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.100787 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" event={"ID":"3751da0e-99b6-4f2c-836e-699267f1428d","Type":"ContainerDied","Data":"4c3b0fc779ee2de7c5544a0b38b6ed949ee52c31b86872cfa62f7d7630d2a951"} Nov 02 23:18:21.100975 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.100856 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7" Nov 02 23:18:21.108484 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.108457 5076 generic.go:356] "Generic (PLEG): container finished" podID="73f3edf1-31bb-41a5-abb8-8a97ab98b1b4" containerID="dba449aa9a5e3ef461583ed7bdd9e17a1ee3f32938ae213fb4a53998b535a5fa" exitCode=0 Nov 02 23:18:21.108630 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.108515 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" Nov 02 23:18:21.108630 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.108539 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" event={"ID":"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4","Type":"ContainerDied","Data":"dba449aa9a5e3ef461583ed7bdd9e17a1ee3f32938ae213fb4a53998b535a5fa"} Nov 02 23:18:21.108630 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.108586 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8" event={"ID":"73f3edf1-31bb-41a5-abb8-8a97ab98b1b4","Type":"ContainerDied","Data":"5db6b1c474e7b3d7541ebfeae476ab86de0e593ba1f6a63ed4900e58f9fcc45d"} Nov 02 23:18:21.110256 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.110236 5076 generic.go:356] "Generic (PLEG): container finished" podID="d18c1e9a-d8c2-4438-8e96-2393552f495e" containerID="9e43b7e56f165e0d0165337b511036ea62356c7a748236f22035f656db077dc2" exitCode=0 Nov 02 23:18:21.110350 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.110320 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" event={"ID":"d18c1e9a-d8c2-4438-8e96-2393552f495e","Type":"ContainerDied","Data":"9e43b7e56f165e0d0165337b511036ea62356c7a748236f22035f656db077dc2"} Nov 02 23:18:21.110496 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.110332 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" Nov 02 23:18:21.110624 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.110356 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" event={"ID":"d18c1e9a-d8c2-4438-8e96-2393552f495e","Type":"ContainerDied","Data":"73ae02e5fb6f93ef56b1a9d57b8990fc5dd88bd1f45f31110795059770526c4a"} Nov 02 23:18:21.112193 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.112175 5076 scope.go:122] "RemoveContainer" containerID="12ea9b1a63208befc24d12c9fa7be2120eaefb2fcaece2475a13f4a73265eed7" Nov 02 23:18:21.112615 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:18:21.112519 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12ea9b1a63208befc24d12c9fa7be2120eaefb2fcaece2475a13f4a73265eed7\": container with ID starting with 12ea9b1a63208befc24d12c9fa7be2120eaefb2fcaece2475a13f4a73265eed7 not found: ID does not exist" containerID="12ea9b1a63208befc24d12c9fa7be2120eaefb2fcaece2475a13f4a73265eed7" Nov 02 23:18:21.112615 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.112552 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12ea9b1a63208befc24d12c9fa7be2120eaefb2fcaece2475a13f4a73265eed7"} err="failed to get container status \"12ea9b1a63208befc24d12c9fa7be2120eaefb2fcaece2475a13f4a73265eed7\": rpc error: code = NotFound desc = could not find container \"12ea9b1a63208befc24d12c9fa7be2120eaefb2fcaece2475a13f4a73265eed7\": container with ID starting with 12ea9b1a63208befc24d12c9fa7be2120eaefb2fcaece2475a13f4a73265eed7 not found: ID does not exist" Nov 02 23:18:21.112615 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.112575 5076 scope.go:122] "RemoveContainer" containerID="9f9e7a539b3f290115ff21f86c5bfe2ade555ff64c7699eee91e34c4b032447c" Nov 02 23:18:21.123349 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.123178 5076 scope.go:122] "RemoveContainer" containerID="9f9e7a539b3f290115ff21f86c5bfe2ade555ff64c7699eee91e34c4b032447c" Nov 02 23:18:21.123461 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:18:21.123442 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f9e7a539b3f290115ff21f86c5bfe2ade555ff64c7699eee91e34c4b032447c\": container with ID starting with 9f9e7a539b3f290115ff21f86c5bfe2ade555ff64c7699eee91e34c4b032447c not found: ID does not exist" containerID="9f9e7a539b3f290115ff21f86c5bfe2ade555ff64c7699eee91e34c4b032447c" Nov 02 23:18:21.123530 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.123472 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f9e7a539b3f290115ff21f86c5bfe2ade555ff64c7699eee91e34c4b032447c"} err="failed to get container status \"9f9e7a539b3f290115ff21f86c5bfe2ade555ff64c7699eee91e34c4b032447c\": rpc error: code = NotFound desc = could not find container \"9f9e7a539b3f290115ff21f86c5bfe2ade555ff64c7699eee91e34c4b032447c\": container with ID starting with 9f9e7a539b3f290115ff21f86c5bfe2ade555ff64c7699eee91e34c4b032447c not found: ID does not exist" Nov 02 23:18:21.123530 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.123498 5076 scope.go:122] "RemoveContainer" containerID="dba449aa9a5e3ef461583ed7bdd9e17a1ee3f32938ae213fb4a53998b535a5fa" Nov 02 23:18:21.139428 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.139403 5076 scope.go:122] "RemoveContainer" containerID="dba449aa9a5e3ef461583ed7bdd9e17a1ee3f32938ae213fb4a53998b535a5fa" Nov 02 23:18:21.146046 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:18:21.146006 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dba449aa9a5e3ef461583ed7bdd9e17a1ee3f32938ae213fb4a53998b535a5fa\": container with ID starting with dba449aa9a5e3ef461583ed7bdd9e17a1ee3f32938ae213fb4a53998b535a5fa not found: ID does not exist" containerID="dba449aa9a5e3ef461583ed7bdd9e17a1ee3f32938ae213fb4a53998b535a5fa" Nov 02 23:18:21.146231 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.146055 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dba449aa9a5e3ef461583ed7bdd9e17a1ee3f32938ae213fb4a53998b535a5fa"} err="failed to get container status \"dba449aa9a5e3ef461583ed7bdd9e17a1ee3f32938ae213fb4a53998b535a5fa\": rpc error: code = NotFound desc = could not find container \"dba449aa9a5e3ef461583ed7bdd9e17a1ee3f32938ae213fb4a53998b535a5fa\": container with ID starting with dba449aa9a5e3ef461583ed7bdd9e17a1ee3f32938ae213fb4a53998b535a5fa not found: ID does not exist" Nov 02 23:18:21.146231 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.146087 5076 scope.go:122] "RemoveContainer" containerID="9e43b7e56f165e0d0165337b511036ea62356c7a748236f22035f656db077dc2" Nov 02 23:18:21.163067 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.163043 5076 scope.go:122] "RemoveContainer" containerID="9e43b7e56f165e0d0165337b511036ea62356c7a748236f22035f656db077dc2" Nov 02 23:18:21.163392 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:18:21.163371 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e43b7e56f165e0d0165337b511036ea62356c7a748236f22035f656db077dc2\": container with ID starting with 9e43b7e56f165e0d0165337b511036ea62356c7a748236f22035f656db077dc2 not found: ID does not exist" containerID="9e43b7e56f165e0d0165337b511036ea62356c7a748236f22035f656db077dc2" Nov 02 23:18:21.163465 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.163401 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e43b7e56f165e0d0165337b511036ea62356c7a748236f22035f656db077dc2"} err="failed to get container status \"9e43b7e56f165e0d0165337b511036ea62356c7a748236f22035f656db077dc2\": rpc error: code = NotFound desc = could not find container \"9e43b7e56f165e0d0165337b511036ea62356c7a748236f22035f656db077dc2\": container with ID starting with 9e43b7e56f165e0d0165337b511036ea62356c7a748236f22035f656db077dc2 not found: ID does not exist" Nov 02 23:18:21.262792 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.262736 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls"] Nov 02 23:18:21.275023 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.274971 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-l96ls"] Nov 02 23:18:21.386660 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.386616 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx"] Nov 02 23:18:21.386855 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.386685 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx"] Nov 02 23:18:21.485961 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.485920 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7"] Nov 02 23:18:21.489684 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.489658 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-t5qb7"] Nov 02 23:18:21.592721 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.592666 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8"] Nov 02 23:18:21.596221 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.596193 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-v6zn8"] Nov 02 23:18:21.693118 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.689123 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:18:21.693118 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:21.689354 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6znpx" podUID="d18c1e9a-d8c2-4438-8e96-2393552f495e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:18:23.149045 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:23.149013 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c2d7a91-9af2-40b7-a010-1a19260cf6f9" path="/var/lib/kubelet/pods/0c2d7a91-9af2-40b7-a010-1a19260cf6f9/volumes" Nov 02 23:18:23.149546 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:23.149526 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3751da0e-99b6-4f2c-836e-699267f1428d" path="/var/lib/kubelet/pods/3751da0e-99b6-4f2c-836e-699267f1428d/volumes" Nov 02 23:18:23.150157 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:23.150138 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73f3edf1-31bb-41a5-abb8-8a97ab98b1b4" path="/var/lib/kubelet/pods/73f3edf1-31bb-41a5-abb8-8a97ab98b1b4/volumes" Nov 02 23:18:23.150946 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:23.150926 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d18c1e9a-d8c2-4438-8e96-2393552f495e" path="/var/lib/kubelet/pods/d18c1e9a-d8c2-4438-8e96-2393552f495e/volumes" Nov 02 23:18:24.031172 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:24.031143 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68e067-7b49597786-b7bwj" Nov 02 23:18:25.630451 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:25.630422 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" Nov 02 23:18:25.683238 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:25.683157 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f19211b1-29a9-4e75-bd8a-ab654254a5a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19211b1-29a9-4e75-bd8a-ab654254a5a1-credential-socket\") pod \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\" (UID: \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\") " Nov 02 23:18:25.683238 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:25.683224 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f19211b1-29a9-4e75-bd8a-ab654254a5a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f19211b1-29a9-4e75-bd8a-ab654254a5a1-istio-envoy\") pod \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\" (UID: \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\") " Nov 02 23:18:25.683469 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:25.683277 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f19211b1-29a9-4e75-bd8a-ab654254a5a1-kube-api-access-dd58n\" (UniqueName: \"kubernetes.io/projected/f19211b1-29a9-4e75-bd8a-ab654254a5a1-kube-api-access-dd58n\") pod \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\" (UID: \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\") " Nov 02 23:18:25.683469 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:25.683364 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f19211b1-29a9-4e75-bd8a-ab654254a5a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19211b1-29a9-4e75-bd8a-ab654254a5a1-workload-socket\") pod \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\" (UID: \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\") " Nov 02 23:18:25.683595 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:25.683491 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f19211b1-29a9-4e75-bd8a-ab654254a5a1-istio-token\" (UniqueName: \"kubernetes.io/projected/f19211b1-29a9-4e75-bd8a-ab654254a5a1-istio-token\") pod \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\" (UID: \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\") " Nov 02 23:18:25.683595 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:25.683506 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19211b1-29a9-4e75-bd8a-ab654254a5a1-credential-socket" pod "f19211b1-29a9-4e75-bd8a-ab654254a5a1" (UID: "f19211b1-29a9-4e75-bd8a-ab654254a5a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:25.683675 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:25.683591 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f19211b1-29a9-4e75-bd8a-ab654254a5a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f19211b1-29a9-4e75-bd8a-ab654254a5a1-istiod-ca-cert\") pod \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\" (UID: \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\") " Nov 02 23:18:25.683711 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:25.683678 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19211b1-29a9-4e75-bd8a-ab654254a5a1-workload-socket" pod "f19211b1-29a9-4e75-bd8a-ab654254a5a1" (UID: "f19211b1-29a9-4e75-bd8a-ab654254a5a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:25.683942 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:25.683627 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f19211b1-29a9-4e75-bd8a-ab654254a5a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f19211b1-29a9-4e75-bd8a-ab654254a5a1-istio-data\") pod \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\" (UID: \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\") " Nov 02 23:18:25.684021 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:25.683957 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19211b1-29a9-4e75-bd8a-ab654254a5a1-istio-data" pod "f19211b1-29a9-4e75-bd8a-ab654254a5a1" (UID: "f19211b1-29a9-4e75-bd8a-ab654254a5a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:25.684021 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:25.683986 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f19211b1-29a9-4e75-bd8a-ab654254a5a1-istiod-ca-cert" pod "f19211b1-29a9-4e75-bd8a-ab654254a5a1" (UID: "f19211b1-29a9-4e75-bd8a-ab654254a5a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:18:25.684021 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:25.684015 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f19211b1-29a9-4e75-bd8a-ab654254a5a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19211b1-29a9-4e75-bd8a-ab654254a5a1-workload-certs\") pod \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\" (UID: \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\") " Nov 02 23:18:25.684157 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:25.684068 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f19211b1-29a9-4e75-bd8a-ab654254a5a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f19211b1-29a9-4e75-bd8a-ab654254a5a1-istio-podinfo\") pod \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\" (UID: \"f19211b1-29a9-4e75-bd8a-ab654254a5a1\") " Nov 02 23:18:25.684292 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:25.684268 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19211b1-29a9-4e75-bd8a-ab654254a5a1-workload-certs" pod "f19211b1-29a9-4e75-bd8a-ab654254a5a1" (UID: "f19211b1-29a9-4e75-bd8a-ab654254a5a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:25.684791 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:25.684765 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19211b1-29a9-4e75-bd8a-ab654254a5a1-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:25.684910 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:25.684796 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19211b1-29a9-4e75-bd8a-ab654254a5a1-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:25.684910 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:25.684814 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19211b1-29a9-4e75-bd8a-ab654254a5a1-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:25.684910 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:25.684829 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f19211b1-29a9-4e75-bd8a-ab654254a5a1-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:25.684910 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:25.684845 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f19211b1-29a9-4e75-bd8a-ab654254a5a1-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:25.686048 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:25.686021 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19211b1-29a9-4e75-bd8a-ab654254a5a1-istio-envoy" pod "f19211b1-29a9-4e75-bd8a-ab654254a5a1" (UID: "f19211b1-29a9-4e75-bd8a-ab654254a5a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:25.686124 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:25.686049 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f19211b1-29a9-4e75-bd8a-ab654254a5a1-kube-api-access-dd58n" pod "f19211b1-29a9-4e75-bd8a-ab654254a5a1" (UID: "f19211b1-29a9-4e75-bd8a-ab654254a5a1"). InnerVolumeSpecName "kube-api-access-dd58n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:18:25.686174 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:25.686150 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f19211b1-29a9-4e75-bd8a-ab654254a5a1-istio-token" pod "f19211b1-29a9-4e75-bd8a-ab654254a5a1" (UID: "f19211b1-29a9-4e75-bd8a-ab654254a5a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:18:25.686291 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:25.686273 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f19211b1-29a9-4e75-bd8a-ab654254a5a1-istio-podinfo" pod "f19211b1-29a9-4e75-bd8a-ab654254a5a1" (UID: "f19211b1-29a9-4e75-bd8a-ab654254a5a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:18:25.786513 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:25.786480 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd58n\" (UniqueName: \"kubernetes.io/projected/f19211b1-29a9-4e75-bd8a-ab654254a5a1-kube-api-access-dd58n\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:25.786513 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:25.786512 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f19211b1-29a9-4e75-bd8a-ab654254a5a1-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:25.786709 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:25.786523 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f19211b1-29a9-4e75-bd8a-ab654254a5a1-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:25.786709 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:25.786533 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f19211b1-29a9-4e75-bd8a-ab654254a5a1-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:26.149383 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:26.149349 5076 generic.go:356] "Generic (PLEG): container finished" podID="f19211b1-29a9-4e75-bd8a-ab654254a5a1" containerID="6e7d7ee86caa0d0a1f695f502d590ab2f7b8913972325d81589d247a5d059ad5" exitCode=0 Nov 02 23:18:26.149549 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:26.149400 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" Nov 02 23:18:26.149549 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:26.149438 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" event={"ID":"f19211b1-29a9-4e75-bd8a-ab654254a5a1","Type":"ContainerDied","Data":"6e7d7ee86caa0d0a1f695f502d590ab2f7b8913972325d81589d247a5d059ad5"} Nov 02 23:18:26.149549 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:26.149491 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5" event={"ID":"f19211b1-29a9-4e75-bd8a-ab654254a5a1","Type":"ContainerDied","Data":"bf753d297afd9a9d2e2dce74f1147b6102d6869ecde26ad44ff6685ab2c92578"} Nov 02 23:18:26.149549 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:26.149515 5076 scope.go:122] "RemoveContainer" containerID="6e7d7ee86caa0d0a1f695f502d590ab2f7b8913972325d81589d247a5d059ad5" Nov 02 23:18:26.160404 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:26.160383 5076 scope.go:122] "RemoveContainer" containerID="9e53c98af73a7e04c5d58dfcbcb20d3f136b8f594b9d2ecd87797920dab7c2b9" Nov 02 23:18:26.169979 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:26.169960 5076 scope.go:122] "RemoveContainer" containerID="6e7d7ee86caa0d0a1f695f502d590ab2f7b8913972325d81589d247a5d059ad5" Nov 02 23:18:26.170276 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:18:26.170256 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e7d7ee86caa0d0a1f695f502d590ab2f7b8913972325d81589d247a5d059ad5\": container with ID starting with 6e7d7ee86caa0d0a1f695f502d590ab2f7b8913972325d81589d247a5d059ad5 not found: ID does not exist" containerID="6e7d7ee86caa0d0a1f695f502d590ab2f7b8913972325d81589d247a5d059ad5" Nov 02 23:18:26.170321 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:26.170286 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e7d7ee86caa0d0a1f695f502d590ab2f7b8913972325d81589d247a5d059ad5"} err="failed to get container status \"6e7d7ee86caa0d0a1f695f502d590ab2f7b8913972325d81589d247a5d059ad5\": rpc error: code = NotFound desc = could not find container \"6e7d7ee86caa0d0a1f695f502d590ab2f7b8913972325d81589d247a5d059ad5\": container with ID starting with 6e7d7ee86caa0d0a1f695f502d590ab2f7b8913972325d81589d247a5d059ad5 not found: ID does not exist" Nov 02 23:18:26.170321 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:26.170306 5076 scope.go:122] "RemoveContainer" containerID="9e53c98af73a7e04c5d58dfcbcb20d3f136b8f594b9d2ecd87797920dab7c2b9" Nov 02 23:18:26.170525 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:18:26.170509 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e53c98af73a7e04c5d58dfcbcb20d3f136b8f594b9d2ecd87797920dab7c2b9\": container with ID starting with 9e53c98af73a7e04c5d58dfcbcb20d3f136b8f594b9d2ecd87797920dab7c2b9 not found: ID does not exist" containerID="9e53c98af73a7e04c5d58dfcbcb20d3f136b8f594b9d2ecd87797920dab7c2b9" Nov 02 23:18:26.170569 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:26.170556 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e53c98af73a7e04c5d58dfcbcb20d3f136b8f594b9d2ecd87797920dab7c2b9"} err="failed to get container status \"9e53c98af73a7e04c5d58dfcbcb20d3f136b8f594b9d2ecd87797920dab7c2b9\": rpc error: code = NotFound desc = could not find container \"9e53c98af73a7e04c5d58dfcbcb20d3f136b8f594b9d2ecd87797920dab7c2b9\": container with ID starting with 9e53c98af73a7e04c5d58dfcbcb20d3f136b8f594b9d2ecd87797920dab7c2b9 not found: ID does not exist" Nov 02 23:18:26.308492 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:26.308454 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5"] Nov 02 23:18:26.355519 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:26.355482 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f46208-openshift-default-55cbf9c9d7-jvvw5"] Nov 02 23:18:27.149608 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:27.149564 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f19211b1-29a9-4e75-bd8a-ab654254a5a1" path="/var/lib/kubelet/pods/f19211b1-29a9-4e75-bd8a-ab654254a5a1/volumes" Nov 02 23:18:39.885021 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:18:39.884970 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:18:44.806223 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:44.806138 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k"] Nov 02 23:18:44.806628 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:44.806356 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" podUID="2cfe1803-0eec-4193-aa8f-05b3699a0ed2" containerName="istio-proxy" containerID="cri-o://2898c23dab897cb166e21b4a5c1a3aac6a2533aa31db6c41b8f3d1762c679314" gracePeriod=30 Nov 02 23:18:44.853600 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:44.853560 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz"] Nov 02 23:18:44.853826 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:44.853802 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" podUID="7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe" containerName="istio-proxy" containerID="cri-o://8a09c2709748659f0f63de0be865e5fb0e44a25a3aeef856e214c5e6b569e360" gracePeriod=30 Nov 02 23:18:44.856854 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:44.856828 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl"] Nov 02 23:18:44.857110 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:44.857065 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" podUID="3c34dfac-f28d-420f-a6da-8279cae95313" containerName="istio-proxy" containerID="cri-o://c254545bd149977bcc76c27131e97046ed111ee0fbbd23160cccec8b213b0685" gracePeriod=30 Nov 02 23:18:44.901371 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:44.901337 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq"] Nov 02 23:18:44.901590 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:44.901563 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" podUID="cc003675-8adf-4c2d-af38-9e3ad5af69f4" containerName="istio-proxy" containerID="cri-o://619f650a70fa12a4f1bce5234bde4b9556d72ed621420323781af0d31a62b115" gracePeriod=30 Nov 02 23:18:45.086126 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:45.086036 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-79385a-6d75566cc6-lqscm"] Nov 02 23:18:45.086782 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:45.086755 5076 cpu_manager.go:414] "containerMap: removing container" podUID="0c2d7a91-9af2-40b7-a010-1a19260cf6f9" containerName="istio-proxy" Nov 02 23:18:45.086782 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:45.086781 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c2d7a91-9af2-40b7-a010-1a19260cf6f9" containerName="istio-proxy" Nov 02 23:18:45.087055 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:45.086812 5076 cpu_manager.go:414] "containerMap: removing container" podUID="3751da0e-99b6-4f2c-836e-699267f1428d" containerName="istio-proxy" Nov 02 23:18:45.087055 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:45.086822 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3751da0e-99b6-4f2c-836e-699267f1428d" containerName="istio-proxy" Nov 02 23:18:45.087055 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:45.086851 5076 cpu_manager.go:414] "containerMap: removing container" podUID="f19211b1-29a9-4e75-bd8a-ab654254a5a1" containerName="istio-proxy" Nov 02 23:18:45.087055 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:45.086858 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f19211b1-29a9-4e75-bd8a-ab654254a5a1" containerName="istio-proxy" Nov 02 23:18:45.087055 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:45.086875 5076 cpu_manager.go:414] "containerMap: removing container" podUID="73f3edf1-31bb-41a5-abb8-8a97ab98b1b4" containerName="istio-proxy" Nov 02 23:18:45.087055 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:45.086883 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73f3edf1-31bb-41a5-abb8-8a97ab98b1b4" containerName="istio-proxy" Nov 02 23:18:45.087055 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:45.086927 5076 cpu_manager.go:414] "containerMap: removing container" podUID="f19211b1-29a9-4e75-bd8a-ab654254a5a1" containerName="istio-proxy" Nov 02 23:18:45.087055 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:45.086935 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f19211b1-29a9-4e75-bd8a-ab654254a5a1" containerName="istio-proxy" Nov 02 23:18:45.087055 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:45.086950 5076 cpu_manager.go:414] "containerMap: removing container" podUID="d18c1e9a-d8c2-4438-8e96-2393552f495e" containerName="istio-proxy" Nov 02 23:18:45.087055 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:45.086957 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d18c1e9a-d8c2-4438-8e96-2393552f495e" containerName="istio-proxy" Nov 02 23:18:45.087392 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:45.087104 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="3751da0e-99b6-4f2c-836e-699267f1428d" containerName="istio-proxy" Nov 02 23:18:45.087392 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:45.087125 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="f19211b1-29a9-4e75-bd8a-ab654254a5a1" containerName="istio-proxy" Nov 02 23:18:45.087392 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:45.087135 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="73f3edf1-31bb-41a5-abb8-8a97ab98b1b4" containerName="istio-proxy" Nov 02 23:18:45.087392 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:45.087149 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="f19211b1-29a9-4e75-bd8a-ab654254a5a1" containerName="istio-proxy" Nov 02 23:18:45.087392 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:45.087160 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="d18c1e9a-d8c2-4438-8e96-2393552f495e" containerName="istio-proxy" Nov 02 23:18:45.087392 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:45.087169 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c2d7a91-9af2-40b7-a010-1a19260cf6f9" containerName="istio-proxy" Nov 02 23:18:45.090981 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:45.090960 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79385a-6d75566cc6-lqscm" Nov 02 23:18:45.141184 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:45.141151 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-79385a-serving-cert\"" Nov 02 23:18:45.154630 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:45.154597 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-79385a-6d75566cc6-lqscm"] Nov 02 23:18:45.190835 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:45.190803 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a9b19ee-8d1b-4629-8a24-f59c6ca897dc-maas-api-tls\") pod \"maas-api-e2e-api-b-79385a-6d75566cc6-lqscm\" (UID: \"0a9b19ee-8d1b-4629-8a24-f59c6ca897dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79385a-6d75566cc6-lqscm" Nov 02 23:18:45.190991 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:45.190839 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw58g\" (UniqueName: \"kubernetes.io/projected/0a9b19ee-8d1b-4629-8a24-f59c6ca897dc-kube-api-access-rw58g\") pod \"maas-api-e2e-api-b-79385a-6d75566cc6-lqscm\" (UID: \"0a9b19ee-8d1b-4629-8a24-f59c6ca897dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79385a-6d75566cc6-lqscm" Nov 02 23:18:45.190991 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:45.190861 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9b19ee-8d1b-4629-8a24-f59c6ca897dc-openshift-service-ca\") pod \"maas-api-e2e-api-b-79385a-6d75566cc6-lqscm\" (UID: \"0a9b19ee-8d1b-4629-8a24-f59c6ca897dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79385a-6d75566cc6-lqscm" Nov 02 23:18:45.191077 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:45.191044 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a9b19ee-8d1b-4629-8a24-f59c6ca897dc-metrics-certs\") pod \"maas-api-e2e-api-b-79385a-6d75566cc6-lqscm\" (UID: \"0a9b19ee-8d1b-4629-8a24-f59c6ca897dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79385a-6d75566cc6-lqscm" Nov 02 23:18:45.292392 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:45.292358 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a9b19ee-8d1b-4629-8a24-f59c6ca897dc-maas-api-tls\") pod \"maas-api-e2e-api-b-79385a-6d75566cc6-lqscm\" (UID: \"0a9b19ee-8d1b-4629-8a24-f59c6ca897dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79385a-6d75566cc6-lqscm" Nov 02 23:18:45.292589 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:45.292401 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw58g\" (UniqueName: \"kubernetes.io/projected/0a9b19ee-8d1b-4629-8a24-f59c6ca897dc-kube-api-access-rw58g\") pod \"maas-api-e2e-api-b-79385a-6d75566cc6-lqscm\" (UID: \"0a9b19ee-8d1b-4629-8a24-f59c6ca897dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79385a-6d75566cc6-lqscm" Nov 02 23:18:45.292589 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:45.292432 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9b19ee-8d1b-4629-8a24-f59c6ca897dc-openshift-service-ca\") pod \"maas-api-e2e-api-b-79385a-6d75566cc6-lqscm\" (UID: \"0a9b19ee-8d1b-4629-8a24-f59c6ca897dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79385a-6d75566cc6-lqscm" Nov 02 23:18:45.292797 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:45.292774 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a9b19ee-8d1b-4629-8a24-f59c6ca897dc-metrics-certs\") pod \"maas-api-e2e-api-b-79385a-6d75566cc6-lqscm\" (UID: \"0a9b19ee-8d1b-4629-8a24-f59c6ca897dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79385a-6d75566cc6-lqscm" Nov 02 23:18:45.292933 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:18:45.292914 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-79385a-serving-cert: secret "maas-api-e2e-api-b-79385a-serving-cert" not found Nov 02 23:18:45.293003 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:18:45.292981 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a9b19ee-8d1b-4629-8a24-f59c6ca897dc-maas-api-tls podName:0a9b19ee-8d1b-4629-8a24-f59c6ca897dc nodeName:}" failed. No retries permitted until 2026-11-02 23:18:45.792967085 +0000 UTC m=+1574.761498847 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0a9b19ee-8d1b-4629-8a24-f59c6ca897dc-maas-api-tls") pod "maas-api-e2e-api-b-79385a-6d75566cc6-lqscm" (UID: "0a9b19ee-8d1b-4629-8a24-f59c6ca897dc") : secret "maas-api-e2e-api-b-79385a-serving-cert" not found Nov 02 23:18:45.293201 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:45.293178 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9b19ee-8d1b-4629-8a24-f59c6ca897dc-openshift-service-ca\") pod \"maas-api-e2e-api-b-79385a-6d75566cc6-lqscm\" (UID: \"0a9b19ee-8d1b-4629-8a24-f59c6ca897dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79385a-6d75566cc6-lqscm" Nov 02 23:18:45.295237 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:45.295213 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a9b19ee-8d1b-4629-8a24-f59c6ca897dc-metrics-certs\") pod \"maas-api-e2e-api-b-79385a-6d75566cc6-lqscm\" (UID: \"0a9b19ee-8d1b-4629-8a24-f59c6ca897dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79385a-6d75566cc6-lqscm" Nov 02 23:18:45.398967 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:45.388877 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw58g\" (UniqueName: \"kubernetes.io/projected/0a9b19ee-8d1b-4629-8a24-f59c6ca897dc-kube-api-access-rw58g\") pod \"maas-api-e2e-api-b-79385a-6d75566cc6-lqscm\" (UID: \"0a9b19ee-8d1b-4629-8a24-f59c6ca897dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79385a-6d75566cc6-lqscm" Nov 02 23:18:45.819425 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:45.811016 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a9b19ee-8d1b-4629-8a24-f59c6ca897dc-maas-api-tls\") pod \"maas-api-e2e-api-b-79385a-6d75566cc6-lqscm\" (UID: \"0a9b19ee-8d1b-4629-8a24-f59c6ca897dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79385a-6d75566cc6-lqscm" Nov 02 23:18:45.839921 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:45.838006 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a9b19ee-8d1b-4629-8a24-f59c6ca897dc-maas-api-tls\") pod \"maas-api-e2e-api-b-79385a-6d75566cc6-lqscm\" (UID: \"0a9b19ee-8d1b-4629-8a24-f59c6ca897dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79385a-6d75566cc6-lqscm" Nov 02 23:18:46.003875 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:46.003844 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79385a-6d75566cc6-lqscm" Nov 02 23:18:46.441934 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:46.441725 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-79385a-6d75566cc6-lqscm"] Nov 02 23:18:46.472534 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:46.470801 5076 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 23:18:47.374265 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:47.374221 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79385a-6d75566cc6-lqscm" event={"ID":"0a9b19ee-8d1b-4629-8a24-f59c6ca897dc","Type":"ContainerStarted","Data":"12140ee5e5f9cc72635a698ee2d2e1147ebb8980b14f8f5d12fc45e1581fd2fb"} Nov 02 23:18:48.433849 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:48.433809 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79385a-6d75566cc6-lqscm" event={"ID":"0a9b19ee-8d1b-4629-8a24-f59c6ca897dc","Type":"ContainerStarted","Data":"b09ffb6b37ce35f6eedfe9db80088c2d0d7f8c3352d70a1882c88320c5be58d2"} Nov 02 23:18:48.435621 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:48.435595 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79385a-6d75566cc6-lqscm" Nov 02 23:18:50.369492 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:50.369459 5076 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 02 23:18:50.370012 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:50.369981 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" podUID="3c34dfac-f28d-420f-a6da-8279cae95313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 02 23:18:51.112702 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.111080 5076 generic.go:356] "Generic (PLEG): container finished" podID="3c34dfac-f28d-420f-a6da-8279cae95313" containerID="c254545bd149977bcc76c27131e97046ed111ee0fbbd23160cccec8b213b0685" exitCode=0 Nov 02 23:18:51.112702 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.111211 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" event={"ID":"3c34dfac-f28d-420f-a6da-8279cae95313","Type":"ContainerDied","Data":"c254545bd149977bcc76c27131e97046ed111ee0fbbd23160cccec8b213b0685"} Nov 02 23:18:51.117438 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.117406 5076 generic.go:356] "Generic (PLEG): container finished" podID="cc003675-8adf-4c2d-af38-9e3ad5af69f4" containerID="619f650a70fa12a4f1bce5234bde4b9556d72ed621420323781af0d31a62b115" exitCode=0 Nov 02 23:18:51.117695 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.117671 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" event={"ID":"cc003675-8adf-4c2d-af38-9e3ad5af69f4","Type":"ContainerDied","Data":"619f650a70fa12a4f1bce5234bde4b9556d72ed621420323781af0d31a62b115"} Nov 02 23:18:51.117844 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.117828 5076 scope.go:122] "RemoveContainer" containerID="8330bd727513640206cf1fe03726eba7c69976ae3a5bdfec0f9e67bd2ce5254d" Nov 02 23:18:51.294144 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.286537 5076 generic.go:356] "Generic (PLEG): container finished" podID="7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe" containerID="8a09c2709748659f0f63de0be865e5fb0e44a25a3aeef856e214c5e6b569e360" exitCode=0 Nov 02 23:18:51.439914 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.438919 5076 generic.go:356] "Generic (PLEG): container finished" podID="2cfe1803-0eec-4193-aa8f-05b3699a0ed2" containerID="2898c23dab897cb166e21b4a5c1a3aac6a2533aa31db6c41b8f3d1762c679314" exitCode=0 Nov 02 23:18:51.764023 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.737930 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" event={"ID":"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe","Type":"ContainerDied","Data":"8a09c2709748659f0f63de0be865e5fb0e44a25a3aeef856e214c5e6b569e360"} Nov 02 23:18:51.764023 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.737998 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" event={"ID":"2cfe1803-0eec-4193-aa8f-05b3699a0ed2","Type":"ContainerDied","Data":"2898c23dab897cb166e21b4a5c1a3aac6a2533aa31db6c41b8f3d1762c679314"} Nov 02 23:18:51.796990 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.788805 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" Nov 02 23:18:51.879267 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.879212 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" Nov 02 23:18:51.908722 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.908678 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-workload-certs\") pod \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\" (UID: \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\") " Nov 02 23:18:51.909307 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.909283 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-istio-envoy\") pod \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\" (UID: \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\") " Nov 02 23:18:51.909515 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.909500 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-workload-certs\") pod \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\" (UID: \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\") " Nov 02 23:18:51.909620 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.909606 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-istio-token\" (UniqueName: \"kubernetes.io/projected/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-istio-token\") pod \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\" (UID: \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\") " Nov 02 23:18:51.909727 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.909714 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-workload-socket\") pod \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\" (UID: \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\") " Nov 02 23:18:51.909850 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.909829 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-workload-socket\") pod \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\" (UID: \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\") " Nov 02 23:18:51.909974 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.909876 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-istiod-ca-cert\") pod \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\" (UID: \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\") " Nov 02 23:18:51.909974 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.909942 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-istiod-ca-cert\") pod \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\" (UID: \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\") " Nov 02 23:18:51.910079 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.909984 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-kube-api-access-4k6fb\" (UniqueName: \"kubernetes.io/projected/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-kube-api-access-4k6fb\") pod \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\" (UID: \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\") " Nov 02 23:18:51.910079 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.910052 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-kube-api-access-7x9t8\" (UniqueName: \"kubernetes.io/projected/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-kube-api-access-7x9t8\") pod \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\" (UID: \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\") " Nov 02 23:18:51.910179 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.910136 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-istio-token\" (UniqueName: \"kubernetes.io/projected/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-istio-token\") pod \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\" (UID: \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\") " Nov 02 23:18:51.910179 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.910172 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-istio-podinfo\") pod \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\" (UID: \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\") " Nov 02 23:18:51.910282 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.910226 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-istio-data\") pod \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\" (UID: \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\") " Nov 02 23:18:51.910282 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.910255 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-istio-envoy\") pod \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\" (UID: \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\") " Nov 02 23:18:51.910381 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.910323 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-credential-socket\") pod \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\" (UID: \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\") " Nov 02 23:18:51.910431 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.910393 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-istio-data\") pod \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\" (UID: \"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe\") " Nov 02 23:18:51.910487 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.910474 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-credential-socket\") pod \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\" (UID: \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\") " Nov 02 23:18:51.910540 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.910515 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-istio-podinfo\") pod \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\" (UID: \"2cfe1803-0eec-4193-aa8f-05b3699a0ed2\") " Nov 02 23:18:51.914641 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.909222 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-workload-certs" pod "7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe" (UID: "7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:51.914641 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.911323 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-workload-socket" pod "7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe" (UID: "7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:51.914641 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.911438 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-workload-socket" pod "2cfe1803-0eec-4193-aa8f-05b3699a0ed2" (UID: "2cfe1803-0eec-4193-aa8f-05b3699a0ed2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:51.914641 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.911633 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-istiod-ca-cert" pod "2cfe1803-0eec-4193-aa8f-05b3699a0ed2" (UID: "2cfe1803-0eec-4193-aa8f-05b3699a0ed2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:18:51.914641 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.911858 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-istio-data" pod "2cfe1803-0eec-4193-aa8f-05b3699a0ed2" (UID: "2cfe1803-0eec-4193-aa8f-05b3699a0ed2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:51.914641 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.912921 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-credential-socket" pod "7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe" (UID: "7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:51.918510 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.918336 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-workload-certs" pod "2cfe1803-0eec-4193-aa8f-05b3699a0ed2" (UID: "2cfe1803-0eec-4193-aa8f-05b3699a0ed2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:51.918841 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.918671 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-istio-data" pod "7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe" (UID: "7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:51.919182 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.918992 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-credential-socket" pod "2cfe1803-0eec-4193-aa8f-05b3699a0ed2" (UID: "2cfe1803-0eec-4193-aa8f-05b3699a0ed2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:51.919480 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.919430 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-istiod-ca-cert" pod "7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe" (UID: "7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:18:51.924102 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.923613 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-istio-podinfo" pod "7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe" (UID: "7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:18:51.924102 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.923682 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-istio-podinfo" pod "2cfe1803-0eec-4193-aa8f-05b3699a0ed2" (UID: "2cfe1803-0eec-4193-aa8f-05b3699a0ed2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:18:51.924102 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.923767 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-istio-token" pod "2cfe1803-0eec-4193-aa8f-05b3699a0ed2" (UID: "2cfe1803-0eec-4193-aa8f-05b3699a0ed2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:18:51.924102 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.923824 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-istio-token" pod "7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe" (UID: "7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:18:51.924102 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.923870 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-istio-envoy" pod "7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe" (UID: "7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:51.924880 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.924845 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-kube-api-access-7x9t8" pod "7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe" (UID: "7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe"). InnerVolumeSpecName "kube-api-access-7x9t8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:18:51.925156 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.925124 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-istio-envoy" pod "2cfe1803-0eec-4193-aa8f-05b3699a0ed2" (UID: "2cfe1803-0eec-4193-aa8f-05b3699a0ed2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:51.925376 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.925353 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-kube-api-access-4k6fb" pod "2cfe1803-0eec-4193-aa8f-05b3699a0ed2" (UID: "2cfe1803-0eec-4193-aa8f-05b3699a0ed2"). InnerVolumeSpecName "kube-api-access-4k6fb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:18:51.957230 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:51.957172 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79385a-6d75566cc6-lqscm" podStartSLOduration=5.952945048 podStartE2EDuration="6.957152715s" totalImagesPullingTime="1.004207667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:18:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:18:48.555377288 +0000 UTC m=+1577.523909073" watchObservedRunningTime="2026-11-02 23:18:51.957152715 +0000 UTC m=+1580.925684500" Nov 02 23:18:52.024569 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.024231 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:52.024569 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.024263 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:52.024569 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.024278 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:52.024569 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.024294 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:52.024569 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.024309 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:52.024569 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.024324 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:52.024569 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.024339 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:52.024569 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.024352 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:52.024569 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.024367 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:52.024569 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.024381 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:52.024569 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.024394 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:52.024569 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.024408 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:52.024569 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.024422 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:52.024569 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.024436 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:52.024569 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.024452 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:52.024569 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.024465 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:52.024569 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.024479 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k6fb\" (UniqueName: \"kubernetes.io/projected/2cfe1803-0eec-4193-aa8f-05b3699a0ed2-kube-api-access-4k6fb\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:52.027258 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.024495 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x9t8\" (UniqueName: \"kubernetes.io/projected/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe-kube-api-access-7x9t8\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:52.077708 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.077680 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" Nov 02 23:18:52.105463 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.105439 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" Nov 02 23:18:52.233142 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.233107 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c34dfac-f28d-420f-a6da-8279cae95313-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c34dfac-f28d-420f-a6da-8279cae95313-istio-podinfo\") pod \"3c34dfac-f28d-420f-a6da-8279cae95313\" (UID: \"3c34dfac-f28d-420f-a6da-8279cae95313\") " Nov 02 23:18:52.233406 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.233383 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c34dfac-f28d-420f-a6da-8279cae95313-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c34dfac-f28d-420f-a6da-8279cae95313-workload-socket\") pod \"3c34dfac-f28d-420f-a6da-8279cae95313\" (UID: \"3c34dfac-f28d-420f-a6da-8279cae95313\") " Nov 02 23:18:52.233570 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.233556 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc003675-8adf-4c2d-af38-9e3ad5af69f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc003675-8adf-4c2d-af38-9e3ad5af69f4-istio-podinfo\") pod \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\" (UID: \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\") " Nov 02 23:18:52.233697 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.233678 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc003675-8adf-4c2d-af38-9e3ad5af69f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc003675-8adf-4c2d-af38-9e3ad5af69f4-workload-socket\") pod \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\" (UID: \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\") " Nov 02 23:18:52.233851 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.233834 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc003675-8adf-4c2d-af38-9e3ad5af69f4-kube-api-access-svzks\" (UniqueName: \"kubernetes.io/projected/cc003675-8adf-4c2d-af38-9e3ad5af69f4-kube-api-access-svzks\") pod \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\" (UID: \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\") " Nov 02 23:18:52.234032 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.234015 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc003675-8adf-4c2d-af38-9e3ad5af69f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc003675-8adf-4c2d-af38-9e3ad5af69f4-istio-envoy\") pod \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\" (UID: \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\") " Nov 02 23:18:52.234166 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.234151 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc003675-8adf-4c2d-af38-9e3ad5af69f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc003675-8adf-4c2d-af38-9e3ad5af69f4-istio-data\") pod \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\" (UID: \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\") " Nov 02 23:18:52.234288 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.234272 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc003675-8adf-4c2d-af38-9e3ad5af69f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc003675-8adf-4c2d-af38-9e3ad5af69f4-workload-certs\") pod \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\" (UID: \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\") " Nov 02 23:18:52.234418 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.234402 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c34dfac-f28d-420f-a6da-8279cae95313-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c34dfac-f28d-420f-a6da-8279cae95313-istiod-ca-cert\") pod \"3c34dfac-f28d-420f-a6da-8279cae95313\" (UID: \"3c34dfac-f28d-420f-a6da-8279cae95313\") " Nov 02 23:18:52.236920 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.234534 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c34dfac-f28d-420f-a6da-8279cae95313-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c34dfac-f28d-420f-a6da-8279cae95313-istio-envoy\") pod \"3c34dfac-f28d-420f-a6da-8279cae95313\" (UID: \"3c34dfac-f28d-420f-a6da-8279cae95313\") " Nov 02 23:18:52.237151 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.237129 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c34dfac-f28d-420f-a6da-8279cae95313-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c34dfac-f28d-420f-a6da-8279cae95313-workload-certs\") pod \"3c34dfac-f28d-420f-a6da-8279cae95313\" (UID: \"3c34dfac-f28d-420f-a6da-8279cae95313\") " Nov 02 23:18:52.237296 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.237277 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc003675-8adf-4c2d-af38-9e3ad5af69f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc003675-8adf-4c2d-af38-9e3ad5af69f4-credential-socket\") pod \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\" (UID: \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\") " Nov 02 23:18:52.237422 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.237404 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc003675-8adf-4c2d-af38-9e3ad5af69f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc003675-8adf-4c2d-af38-9e3ad5af69f4-istiod-ca-cert\") pod \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\" (UID: \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\") " Nov 02 23:18:52.237572 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.237555 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc003675-8adf-4c2d-af38-9e3ad5af69f4-istio-token\" (UniqueName: \"kubernetes.io/projected/cc003675-8adf-4c2d-af38-9e3ad5af69f4-istio-token\") pod \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\" (UID: \"cc003675-8adf-4c2d-af38-9e3ad5af69f4\") " Nov 02 23:18:52.240465 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.240442 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c34dfac-f28d-420f-a6da-8279cae95313-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c34dfac-f28d-420f-a6da-8279cae95313-credential-socket\") pod \"3c34dfac-f28d-420f-a6da-8279cae95313\" (UID: \"3c34dfac-f28d-420f-a6da-8279cae95313\") " Nov 02 23:18:52.240649 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.240633 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c34dfac-f28d-420f-a6da-8279cae95313-kube-api-access-mkgsw\" (UniqueName: \"kubernetes.io/projected/3c34dfac-f28d-420f-a6da-8279cae95313-kube-api-access-mkgsw\") pod \"3c34dfac-f28d-420f-a6da-8279cae95313\" (UID: \"3c34dfac-f28d-420f-a6da-8279cae95313\") " Nov 02 23:18:52.240792 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.240778 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c34dfac-f28d-420f-a6da-8279cae95313-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c34dfac-f28d-420f-a6da-8279cae95313-istio-data\") pod \"3c34dfac-f28d-420f-a6da-8279cae95313\" (UID: \"3c34dfac-f28d-420f-a6da-8279cae95313\") " Nov 02 23:18:52.240983 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.240967 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c34dfac-f28d-420f-a6da-8279cae95313-istio-token\" (UniqueName: \"kubernetes.io/projected/3c34dfac-f28d-420f-a6da-8279cae95313-istio-token\") pod \"3c34dfac-f28d-420f-a6da-8279cae95313\" (UID: \"3c34dfac-f28d-420f-a6da-8279cae95313\") " Nov 02 23:18:52.241422 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.241398 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c34dfac-f28d-420f-a6da-8279cae95313-credential-socket" pod "3c34dfac-f28d-420f-a6da-8279cae95313" (UID: "3c34dfac-f28d-420f-a6da-8279cae95313"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:52.241580 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.241554 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc003675-8adf-4c2d-af38-9e3ad5af69f4-istio-data" pod "cc003675-8adf-4c2d-af38-9e3ad5af69f4" (UID: "cc003675-8adf-4c2d-af38-9e3ad5af69f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:52.242342 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.242171 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc003675-8adf-4c2d-af38-9e3ad5af69f4-workload-certs" pod "cc003675-8adf-4c2d-af38-9e3ad5af69f4" (UID: "cc003675-8adf-4c2d-af38-9e3ad5af69f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:52.242342 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.242202 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc003675-8adf-4c2d-af38-9e3ad5af69f4-workload-socket" pod "cc003675-8adf-4c2d-af38-9e3ad5af69f4" (UID: "cc003675-8adf-4c2d-af38-9e3ad5af69f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:52.242342 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.242277 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc003675-8adf-4c2d-af38-9e3ad5af69f4-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:52.242342 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.242320 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c34dfac-f28d-420f-a6da-8279cae95313-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:52.242516 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.242397 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c34dfac-f28d-420f-a6da-8279cae95313-istio-data" pod "3c34dfac-f28d-420f-a6da-8279cae95313" (UID: "3c34dfac-f28d-420f-a6da-8279cae95313"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:52.242516 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.242413 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc003675-8adf-4c2d-af38-9e3ad5af69f4-istio-token" pod "cc003675-8adf-4c2d-af38-9e3ad5af69f4" (UID: "cc003675-8adf-4c2d-af38-9e3ad5af69f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:18:52.243035 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.242405 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c34dfac-f28d-420f-a6da-8279cae95313-workload-socket" pod "3c34dfac-f28d-420f-a6da-8279cae95313" (UID: "3c34dfac-f28d-420f-a6da-8279cae95313"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:52.243035 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.243024 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc003675-8adf-4c2d-af38-9e3ad5af69f4-istiod-ca-cert" pod "cc003675-8adf-4c2d-af38-9e3ad5af69f4" (UID: "cc003675-8adf-4c2d-af38-9e3ad5af69f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:18:52.243158 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.242635 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c34dfac-f28d-420f-a6da-8279cae95313-workload-certs" pod "3c34dfac-f28d-420f-a6da-8279cae95313" (UID: "3c34dfac-f28d-420f-a6da-8279cae95313"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:52.243158 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.242747 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc003675-8adf-4c2d-af38-9e3ad5af69f4-credential-socket" pod "cc003675-8adf-4c2d-af38-9e3ad5af69f4" (UID: "cc003675-8adf-4c2d-af38-9e3ad5af69f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:52.243158 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.242777 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c34dfac-f28d-420f-a6da-8279cae95313-istiod-ca-cert" pod "3c34dfac-f28d-420f-a6da-8279cae95313" (UID: "3c34dfac-f28d-420f-a6da-8279cae95313"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:18:52.244878 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.244848 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c34dfac-f28d-420f-a6da-8279cae95313-istio-token" pod "3c34dfac-f28d-420f-a6da-8279cae95313" (UID: "3c34dfac-f28d-420f-a6da-8279cae95313"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:18:52.245089 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.244866 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c34dfac-f28d-420f-a6da-8279cae95313-kube-api-access-mkgsw" pod "3c34dfac-f28d-420f-a6da-8279cae95313" (UID: "3c34dfac-f28d-420f-a6da-8279cae95313"). InnerVolumeSpecName "kube-api-access-mkgsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:18:52.245309 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.245273 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c34dfac-f28d-420f-a6da-8279cae95313-istio-envoy" pod "3c34dfac-f28d-420f-a6da-8279cae95313" (UID: "3c34dfac-f28d-420f-a6da-8279cae95313"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:52.245538 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.245511 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc003675-8adf-4c2d-af38-9e3ad5af69f4-istio-envoy" pod "cc003675-8adf-4c2d-af38-9e3ad5af69f4" (UID: "cc003675-8adf-4c2d-af38-9e3ad5af69f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:52.246117 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.246089 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c34dfac-f28d-420f-a6da-8279cae95313-istio-podinfo" pod "3c34dfac-f28d-420f-a6da-8279cae95313" (UID: "3c34dfac-f28d-420f-a6da-8279cae95313"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:18:52.246220 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.246183 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc003675-8adf-4c2d-af38-9e3ad5af69f4-istio-podinfo" pod "cc003675-8adf-4c2d-af38-9e3ad5af69f4" (UID: "cc003675-8adf-4c2d-af38-9e3ad5af69f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:18:52.246724 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.246696 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc003675-8adf-4c2d-af38-9e3ad5af69f4-kube-api-access-svzks" pod "cc003675-8adf-4c2d-af38-9e3ad5af69f4" (UID: "cc003675-8adf-4c2d-af38-9e3ad5af69f4"). InnerVolumeSpecName "kube-api-access-svzks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:18:52.352039 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.348027 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c34dfac-f28d-420f-a6da-8279cae95313-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:52.352039 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.348065 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c34dfac-f28d-420f-a6da-8279cae95313-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:52.352039 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.348082 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc003675-8adf-4c2d-af38-9e3ad5af69f4-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:52.352039 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.348096 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc003675-8adf-4c2d-af38-9e3ad5af69f4-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:52.352039 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.348111 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svzks\" (UniqueName: \"kubernetes.io/projected/cc003675-8adf-4c2d-af38-9e3ad5af69f4-kube-api-access-svzks\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:52.352039 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.348129 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc003675-8adf-4c2d-af38-9e3ad5af69f4-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:52.352039 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.348145 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc003675-8adf-4c2d-af38-9e3ad5af69f4-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:52.352039 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.348159 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c34dfac-f28d-420f-a6da-8279cae95313-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:52.352039 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.348173 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c34dfac-f28d-420f-a6da-8279cae95313-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:52.352039 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.348187 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c34dfac-f28d-420f-a6da-8279cae95313-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:52.352039 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.348201 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc003675-8adf-4c2d-af38-9e3ad5af69f4-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:52.352039 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.348214 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc003675-8adf-4c2d-af38-9e3ad5af69f4-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:52.352039 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.348228 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc003675-8adf-4c2d-af38-9e3ad5af69f4-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:52.352039 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.348242 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkgsw\" (UniqueName: \"kubernetes.io/projected/3c34dfac-f28d-420f-a6da-8279cae95313-kube-api-access-mkgsw\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:52.352039 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.348255 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c34dfac-f28d-420f-a6da-8279cae95313-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:52.352039 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.348270 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c34dfac-f28d-420f-a6da-8279cae95313-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:52.451872 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.451837 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" Nov 02 23:18:52.452336 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.451841 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz" event={"ID":"7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe","Type":"ContainerDied","Data":"591d35ab3f12a96e87235d6ba6d85528300cea12756e5c7cb1d7238c3cd1620a"} Nov 02 23:18:52.452336 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.452020 5076 scope.go:122] "RemoveContainer" containerID="8a09c2709748659f0f63de0be865e5fb0e44a25a3aeef856e214c5e6b569e360" Nov 02 23:18:52.453674 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.453652 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" Nov 02 23:18:52.453674 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.453656 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k" event={"ID":"2cfe1803-0eec-4193-aa8f-05b3699a0ed2","Type":"ContainerDied","Data":"dac71ce00b193f064b47ec110ce7f4c2e1c8c0b00fb731172d20516652bb9566"} Nov 02 23:18:52.455369 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.455350 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" Nov 02 23:18:52.455369 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.455352 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl" event={"ID":"3c34dfac-f28d-420f-a6da-8279cae95313","Type":"ContainerDied","Data":"df1687a185d743959f1035cc4688a977bb0cd7fb2830c08883f46538b57cd2a5"} Nov 02 23:18:52.457518 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.457474 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" event={"ID":"cc003675-8adf-4c2d-af38-9e3ad5af69f4","Type":"ContainerDied","Data":"eeda88117570750d051fb6cf5b1314a486a2bc1dc1d502d67a80620112e3e193"} Nov 02 23:18:52.457518 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.457510 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq" Nov 02 23:18:52.463067 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.463056 5076 scope.go:122] "RemoveContainer" containerID="2898c23dab897cb166e21b4a5c1a3aac6a2533aa31db6c41b8f3d1762c679314" Nov 02 23:18:52.473926 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.473879 5076 scope.go:122] "RemoveContainer" containerID="c254545bd149977bcc76c27131e97046ed111ee0fbbd23160cccec8b213b0685" Nov 02 23:18:52.482795 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.482775 5076 scope.go:122] "RemoveContainer" containerID="619f650a70fa12a4f1bce5234bde4b9556d72ed621420323781af0d31a62b115" Nov 02 23:18:52.628970 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.610059 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k"] Nov 02 23:18:52.637157 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.637120 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-n6t6k"] Nov 02 23:18:52.736588 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.736556 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq"] Nov 02 23:18:52.740065 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.740031 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-97lkq"] Nov 02 23:18:52.856468 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.856433 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz"] Nov 02 23:18:52.859774 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.859750 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-7zwlz"] Nov 02 23:18:52.988534 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.988491 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl"] Nov 02 23:18:52.995903 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:52.995868 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-d4wxl"] Nov 02 23:18:53.149114 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:53.149081 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cfe1803-0eec-4193-aa8f-05b3699a0ed2" path="/var/lib/kubelet/pods/2cfe1803-0eec-4193-aa8f-05b3699a0ed2/volumes" Nov 02 23:18:53.149593 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:53.149579 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c34dfac-f28d-420f-a6da-8279cae95313" path="/var/lib/kubelet/pods/3c34dfac-f28d-420f-a6da-8279cae95313/volumes" Nov 02 23:18:53.150018 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:53.150004 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe" path="/var/lib/kubelet/pods/7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe/volumes" Nov 02 23:18:53.150726 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:53.150711 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc003675-8adf-4c2d-af38-9e3ad5af69f4" path="/var/lib/kubelet/pods/cc003675-8adf-4c2d-af38-9e3ad5af69f4/volumes" Nov 02 23:18:54.946259 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:54.946223 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62"] Nov 02 23:18:54.947248 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:54.947224 5076 cpu_manager.go:414] "containerMap: removing container" podUID="7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe" containerName="istio-proxy" Nov 02 23:18:54.947320 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:54.947249 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe" containerName="istio-proxy" Nov 02 23:18:54.947320 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:54.947271 5076 cpu_manager.go:414] "containerMap: removing container" podUID="3c34dfac-f28d-420f-a6da-8279cae95313" containerName="istio-proxy" Nov 02 23:18:54.947320 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:54.947280 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c34dfac-f28d-420f-a6da-8279cae95313" containerName="istio-proxy" Nov 02 23:18:54.947320 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:54.947313 5076 cpu_manager.go:414] "containerMap: removing container" podUID="cc003675-8adf-4c2d-af38-9e3ad5af69f4" containerName="istio-proxy" Nov 02 23:18:54.947475 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:54.947323 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc003675-8adf-4c2d-af38-9e3ad5af69f4" containerName="istio-proxy" Nov 02 23:18:54.947475 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:54.947345 5076 cpu_manager.go:414] "containerMap: removing container" podUID="2cfe1803-0eec-4193-aa8f-05b3699a0ed2" containerName="istio-proxy" Nov 02 23:18:54.947475 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:54.947354 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cfe1803-0eec-4193-aa8f-05b3699a0ed2" containerName="istio-proxy" Nov 02 23:18:54.947585 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:54.947523 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f9a57c0-49f1-435b-8c3b-d64bfe5a1efe" containerName="istio-proxy" Nov 02 23:18:54.947585 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:54.947538 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c34dfac-f28d-420f-a6da-8279cae95313" containerName="istio-proxy" Nov 02 23:18:54.947585 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:54.947552 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc003675-8adf-4c2d-af38-9e3ad5af69f4" containerName="istio-proxy" Nov 02 23:18:54.947585 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:54.947562 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cfe1803-0eec-4193-aa8f-05b3699a0ed2" containerName="istio-proxy" Nov 02 23:18:54.947585 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:54.947575 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc003675-8adf-4c2d-af38-9e3ad5af69f4" containerName="istio-proxy" Nov 02 23:18:54.952004 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:54.951980 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" Nov 02 23:18:55.000913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:54.992706 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801851c2-20cb-4816-9a90-032a209b4933-istio-podinfo\") pod \"e2e-ipp-a-027689-openshift-default-749945876c-n6p62\" (UID: \"801851c2-20cb-4816-9a90-032a209b4933\") " pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" Nov 02 23:18:55.000913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:54.992773 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801851c2-20cb-4816-9a90-032a209b4933-istio-envoy\") pod \"e2e-ipp-a-027689-openshift-default-749945876c-n6p62\" (UID: \"801851c2-20cb-4816-9a90-032a209b4933\") " pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" Nov 02 23:18:55.000913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:54.992845 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801851c2-20cb-4816-9a90-032a209b4933-istiod-ca-cert\") pod \"e2e-ipp-a-027689-openshift-default-749945876c-n6p62\" (UID: \"801851c2-20cb-4816-9a90-032a209b4933\") " pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" Nov 02 23:18:55.000913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:54.992971 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801851c2-20cb-4816-9a90-032a209b4933-credential-socket\") pod \"e2e-ipp-a-027689-openshift-default-749945876c-n6p62\" (UID: \"801851c2-20cb-4816-9a90-032a209b4933\") " pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" Nov 02 23:18:55.000913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:54.993021 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801851c2-20cb-4816-9a90-032a209b4933-istio-data\") pod \"e2e-ipp-a-027689-openshift-default-749945876c-n6p62\" (UID: \"801851c2-20cb-4816-9a90-032a209b4933\") " pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" Nov 02 23:18:55.000913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:54.993055 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/801851c2-20cb-4816-9a90-032a209b4933-istio-token\") pod \"e2e-ipp-a-027689-openshift-default-749945876c-n6p62\" (UID: \"801851c2-20cb-4816-9a90-032a209b4933\") " pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" Nov 02 23:18:55.000913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:54.993080 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801851c2-20cb-4816-9a90-032a209b4933-workload-socket\") pod \"e2e-ipp-a-027689-openshift-default-749945876c-n6p62\" (UID: \"801851c2-20cb-4816-9a90-032a209b4933\") " pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" Nov 02 23:18:55.000913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:54.993218 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801851c2-20cb-4816-9a90-032a209b4933-workload-certs\") pod \"e2e-ipp-a-027689-openshift-default-749945876c-n6p62\" (UID: \"801851c2-20cb-4816-9a90-032a209b4933\") " pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" Nov 02 23:18:55.000913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:54.993244 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rvfc\" (UniqueName: \"kubernetes.io/projected/801851c2-20cb-4816-9a90-032a209b4933-kube-api-access-7rvfc\") pod \"e2e-ipp-a-027689-openshift-default-749945876c-n6p62\" (UID: \"801851c2-20cb-4816-9a90-032a209b4933\") " pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" Nov 02 23:18:55.063914 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:55.063282 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-027689-openshift-default-dockercfg-c54df\"" Nov 02 23:18:55.098943 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:55.097080 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801851c2-20cb-4816-9a90-032a209b4933-workload-socket\") pod \"e2e-ipp-a-027689-openshift-default-749945876c-n6p62\" (UID: \"801851c2-20cb-4816-9a90-032a209b4933\") " pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" Nov 02 23:18:55.098943 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:55.097246 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801851c2-20cb-4816-9a90-032a209b4933-workload-certs\") pod \"e2e-ipp-a-027689-openshift-default-749945876c-n6p62\" (UID: \"801851c2-20cb-4816-9a90-032a209b4933\") " pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" Nov 02 23:18:55.098943 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:55.097303 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rvfc\" (UniqueName: \"kubernetes.io/projected/801851c2-20cb-4816-9a90-032a209b4933-kube-api-access-7rvfc\") pod \"e2e-ipp-a-027689-openshift-default-749945876c-n6p62\" (UID: \"801851c2-20cb-4816-9a90-032a209b4933\") " pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" Nov 02 23:18:55.098943 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:55.097367 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801851c2-20cb-4816-9a90-032a209b4933-istio-podinfo\") pod \"e2e-ipp-a-027689-openshift-default-749945876c-n6p62\" (UID: \"801851c2-20cb-4816-9a90-032a209b4933\") " pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" Nov 02 23:18:55.098943 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:55.097403 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801851c2-20cb-4816-9a90-032a209b4933-istio-envoy\") pod \"e2e-ipp-a-027689-openshift-default-749945876c-n6p62\" (UID: \"801851c2-20cb-4816-9a90-032a209b4933\") " pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" Nov 02 23:18:55.098943 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:55.097484 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801851c2-20cb-4816-9a90-032a209b4933-istiod-ca-cert\") pod \"e2e-ipp-a-027689-openshift-default-749945876c-n6p62\" (UID: \"801851c2-20cb-4816-9a90-032a209b4933\") " pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" Nov 02 23:18:55.098943 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:55.097588 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801851c2-20cb-4816-9a90-032a209b4933-credential-socket\") pod \"e2e-ipp-a-027689-openshift-default-749945876c-n6p62\" (UID: \"801851c2-20cb-4816-9a90-032a209b4933\") " pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" Nov 02 23:18:55.098943 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:55.097633 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801851c2-20cb-4816-9a90-032a209b4933-istio-data\") pod \"e2e-ipp-a-027689-openshift-default-749945876c-n6p62\" (UID: \"801851c2-20cb-4816-9a90-032a209b4933\") " pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" Nov 02 23:18:55.098943 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:55.097664 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/801851c2-20cb-4816-9a90-032a209b4933-istio-token\") pod \"e2e-ipp-a-027689-openshift-default-749945876c-n6p62\" (UID: \"801851c2-20cb-4816-9a90-032a209b4933\") " pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" Nov 02 23:18:55.098943 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:55.098397 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801851c2-20cb-4816-9a90-032a209b4933-workload-socket\") pod \"e2e-ipp-a-027689-openshift-default-749945876c-n6p62\" (UID: \"801851c2-20cb-4816-9a90-032a209b4933\") " pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" Nov 02 23:18:55.098943 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:55.098636 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801851c2-20cb-4816-9a90-032a209b4933-workload-certs\") pod \"e2e-ipp-a-027689-openshift-default-749945876c-n6p62\" (UID: \"801851c2-20cb-4816-9a90-032a209b4933\") " pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" Nov 02 23:18:55.118388 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:55.117078 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801851c2-20cb-4816-9a90-032a209b4933-istiod-ca-cert\") pod \"e2e-ipp-a-027689-openshift-default-749945876c-n6p62\" (UID: \"801851c2-20cb-4816-9a90-032a209b4933\") " pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" Nov 02 23:18:55.138255 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:55.133218 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62"] Nov 02 23:18:55.138255 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:55.133667 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801851c2-20cb-4816-9a90-032a209b4933-credential-socket\") pod \"e2e-ipp-a-027689-openshift-default-749945876c-n6p62\" (UID: \"801851c2-20cb-4816-9a90-032a209b4933\") " pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" Nov 02 23:18:55.138255 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:55.133934 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801851c2-20cb-4816-9a90-032a209b4933-istio-data\") pod \"e2e-ipp-a-027689-openshift-default-749945876c-n6p62\" (UID: \"801851c2-20cb-4816-9a90-032a209b4933\") " pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" Nov 02 23:18:55.139592 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:55.139515 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801851c2-20cb-4816-9a90-032a209b4933-istio-podinfo\") pod \"e2e-ipp-a-027689-openshift-default-749945876c-n6p62\" (UID: \"801851c2-20cb-4816-9a90-032a209b4933\") " pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" Nov 02 23:18:55.181380 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:55.178102 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801851c2-20cb-4816-9a90-032a209b4933-istio-envoy\") pod \"e2e-ipp-a-027689-openshift-default-749945876c-n6p62\" (UID: \"801851c2-20cb-4816-9a90-032a209b4933\") " pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" Nov 02 23:18:55.292508 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:55.292365 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rvfc\" (UniqueName: \"kubernetes.io/projected/801851c2-20cb-4816-9a90-032a209b4933-kube-api-access-7rvfc\") pod \"e2e-ipp-a-027689-openshift-default-749945876c-n6p62\" (UID: \"801851c2-20cb-4816-9a90-032a209b4933\") " pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" Nov 02 23:18:55.314088 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:55.313434 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/801851c2-20cb-4816-9a90-032a209b4933-istio-token\") pod \"e2e-ipp-a-027689-openshift-default-749945876c-n6p62\" (UID: \"801851c2-20cb-4816-9a90-032a209b4933\") " pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" Nov 02 23:18:55.426431 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:55.426390 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8"] Nov 02 23:18:55.427047 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:55.427027 5076 cpu_manager.go:414] "containerMap: removing container" podUID="cc003675-8adf-4c2d-af38-9e3ad5af69f4" containerName="istio-proxy" Nov 02 23:18:55.427047 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:55.427044 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc003675-8adf-4c2d-af38-9e3ad5af69f4" containerName="istio-proxy" Nov 02 23:18:55.430186 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:55.430167 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8" Nov 02 23:18:55.518791 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:55.498686 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e953257b-serving-cert\"" Nov 02 23:18:55.525336 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:55.525303 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8"] Nov 02 23:18:55.568919 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:55.568794 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" Nov 02 23:18:55.571476 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:55.571358 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb6z9\" (UniqueName: \"kubernetes.io/projected/fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7-kube-api-access-fb6z9\") pod \"maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8\" (UID: \"fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8" Nov 02 23:18:55.571476 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:55.571425 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7-maas-api-tls\") pod \"maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8\" (UID: \"fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8" Nov 02 23:18:55.571476 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:55.571452 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7-metrics-certs\") pod \"maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8\" (UID: \"fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8" Nov 02 23:18:55.571674 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:55.571602 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7-openshift-service-ca\") pod \"maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8\" (UID: \"fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8" Nov 02 23:18:55.674921 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:55.673774 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7-openshift-service-ca\") pod \"maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8\" (UID: \"fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8" Nov 02 23:18:55.674921 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:55.673940 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb6z9\" (UniqueName: \"kubernetes.io/projected/fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7-kube-api-access-fb6z9\") pod \"maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8\" (UID: \"fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8" Nov 02 23:18:55.674921 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:55.674013 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7-maas-api-tls\") pod \"maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8\" (UID: \"fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8" Nov 02 23:18:55.674921 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:55.674042 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7-metrics-certs\") pod \"maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8\" (UID: \"fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8" Nov 02 23:18:55.674921 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:55.674750 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7-openshift-service-ca\") pod \"maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8\" (UID: \"fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8" Nov 02 23:18:55.674921 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:18:55.674866 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-serving-cert: secret "maas-api-e2e-mt-e953257b-serving-cert" not found Nov 02 23:18:55.675348 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:18:55.675039 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7-maas-api-tls podName:fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7 nodeName:}" failed. No retries permitted until 2026-11-02 23:18:56.175019004 +0000 UTC m=+1585.143550785 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7-maas-api-tls") pod "maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8" (UID: "fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7") : secret "maas-api-e2e-mt-e953257b-serving-cert" not found Nov 02 23:18:55.742798 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:55.742731 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb6z9\" (UniqueName: \"kubernetes.io/projected/fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7-kube-api-access-fb6z9\") pod \"maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8\" (UID: \"fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8" Nov 02 23:18:55.770909 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:55.768675 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7-metrics-certs\") pod \"maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8\" (UID: \"fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8" Nov 02 23:18:56.027122 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:56.027088 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62"] Nov 02 23:18:56.028766 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:18:56.028738 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod801851c2_20cb_4816_9a90_032a209b4933.slice/crio-b114d5ff15c3ae93edac81f757b51a389f533171c8324225930324c8e705ac87 WatchSource:0}: Error finding container b114d5ff15c3ae93edac81f757b51a389f533171c8324225930324c8e705ac87: Status 404 returned error can't find the container with id b114d5ff15c3ae93edac81f757b51a389f533171c8324225930324c8e705ac87 Nov 02 23:18:56.031777 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:56.031749 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:18:56.031835 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:56.031812 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:18:56.031883 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:56.031842 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:18:56.184872 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:56.184833 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7-maas-api-tls\") pod \"maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8\" (UID: \"fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8" Nov 02 23:18:56.187594 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:56.187563 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7-maas-api-tls\") pod \"maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8\" (UID: \"fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8" Nov 02 23:18:56.346677 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:56.346558 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8" Nov 02 23:18:56.537344 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:56.537298 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" event={"ID":"801851c2-20cb-4816-9a90-032a209b4933","Type":"ContainerStarted","Data":"b9fdc573e3d71a04eddb008d6f94ca045c652354c2f97f4f60d06081b0adf7ce"} Nov 02 23:18:56.537510 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:56.537367 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" event={"ID":"801851c2-20cb-4816-9a90-032a209b4933","Type":"ContainerStarted","Data":"b114d5ff15c3ae93edac81f757b51a389f533171c8324225930324c8e705ac87"} Nov 02 23:18:56.569978 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:56.569943 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" Nov 02 23:18:56.651348 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:56.651235 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" podStartSLOduration=2.651218675 podStartE2EDuration="2.651218675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:18:56.647802398 +0000 UTC m=+1585.616334182" watchObservedRunningTime="2026-11-02 23:18:56.651218675 +0000 UTC m=+1585.619750459" Nov 02 23:18:56.744112 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:56.744080 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79385a-6d75566cc6-lqscm" Nov 02 23:18:56.838870 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:56.838839 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8"] Nov 02 23:18:57.546368 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:57.546327 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8" event={"ID":"fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7","Type":"ContainerStarted","Data":"8f4d55421f317ad74fbf7e63cce73b1d2fd91c3fc4c1cf39f4a2778505cfc81f"} Nov 02 23:18:57.575746 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:57.575709 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" Nov 02 23:18:58.619855 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:58.619809 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8" event={"ID":"fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7","Type":"ContainerStarted","Data":"8338582e008b6c98f9f04b58530758c1666cc13a57f7ee57ff512271f87346ee"} Nov 02 23:18:58.620952 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:58.620845 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8" Nov 02 23:18:58.646077 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:58.646038 5076 generic.go:356] "Generic (PLEG): container finished" podID="e055ddbc-25c2-439e-85a6-b16c23688888" containerID="bb00f3e50511a389808878d36acc49d0ce70589eb29effdc2dee1cf0ce556f2e" exitCode=0 Nov 02 23:18:58.647592 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:58.647558 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" event={"ID":"e055ddbc-25c2-439e-85a6-b16c23688888","Type":"ContainerDied","Data":"bb00f3e50511a389808878d36acc49d0ce70589eb29effdc2dee1cf0ce556f2e"} Nov 02 23:18:58.647838 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:58.647819 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" Nov 02 23:18:58.648557 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:58.648536 5076 scope.go:122] "RemoveContainer" containerID="bb00f3e50511a389808878d36acc49d0ce70589eb29effdc2dee1cf0ce556f2e" Nov 02 23:18:58.648557 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:58.648549 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" Nov 02 23:18:58.650124 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:58.650093 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:18:58.650226 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:58.650170 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:18:58.650226 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:58.650202 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:18:58.733088 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:58.733038 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8" podStartSLOduration=2.853782868 podStartE2EDuration="3.733013661s" totalImagesPullingTime="879.230793ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:18:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:18:58.729311076 +0000 UTC m=+1587.697842862" watchObservedRunningTime="2026-11-02 23:18:58.733013661 +0000 UTC m=+1587.701545444" Nov 02 23:18:59.654365 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:59.654307 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" event={"ID":"e055ddbc-25c2-439e-85a6-b16c23688888","Type":"ContainerStarted","Data":"d1a23f72caa5a553b124a859897ef5ca4d0cc0e65d4e2ca96363c403e48eba3d"} Nov 02 23:18:59.821717 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:59.821620 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" Nov 02 23:18:59.826171 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:59.825804 5076 patch_prober.go:29] interesting pod/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 02 23:18:59.826171 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:18:59.825903 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" podUID="e055ddbc-25c2-439e-85a6-b16c23688888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 02 23:19:00.822048 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:00.822010 5076 patch_prober.go:29] interesting pod/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 02 23:19:00.823110 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:00.823077 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" podUID="e055ddbc-25c2-439e-85a6-b16c23688888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 02 23:19:01.825914 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:01.820305 5076 patch_prober.go:29] interesting pod/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 02 23:19:01.825914 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:01.820370 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" podUID="e055ddbc-25c2-439e-85a6-b16c23688888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 02 23:19:02.101202 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:02.101120 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" Nov 02 23:19:02.834100 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:02.834019 5076 patch_prober.go:29] interesting pod/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 02 23:19:02.834605 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:02.834123 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" podUID="e055ddbc-25c2-439e-85a6-b16c23688888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 02 23:19:03.823343 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:03.819390 5076 patch_prober.go:29] interesting pod/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 02 23:19:03.823343 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:03.819462 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" podUID="e055ddbc-25c2-439e-85a6-b16c23688888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 02 23:19:04.599586 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:04.589824 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-f647578d7-4jc4f"] Nov 02 23:19:04.647214 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:04.647181 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-f647578d7-4jc4f" Nov 02 23:19:04.707916 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:04.699491 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-f647578d7-4jc4f"] Nov 02 23:19:04.789255 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:04.789220 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-027689\"" Nov 02 23:19:04.789564 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:04.789543 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-027689-dockercfg-zrtjl\"" Nov 02 23:19:04.863557 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:04.861044 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whxbj\" (UniqueName: \"kubernetes.io/projected/dd2370ae-3618-4464-939d-1821c4af19bd-kube-api-access-whxbj\") pod \"payload-pre-processing-e2e-ipp-a-027689-f647578d7-4jc4f\" (UID: \"dd2370ae-3618-4464-939d-1821c4af19bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-f647578d7-4jc4f" Nov 02 23:19:04.863941 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:04.863909 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2370ae-3618-4464-939d-1821c4af19bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-027689-f647578d7-4jc4f\" (UID: \"dd2370ae-3618-4464-939d-1821c4af19bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-f647578d7-4jc4f" Nov 02 23:19:04.866267 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:04.866239 5076 patch_prober.go:29] interesting pod/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 02 23:19:04.902042 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:04.901993 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" podUID="e055ddbc-25c2-439e-85a6-b16c23688888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 02 23:19:04.930220 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:04.910364 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-027689-5b9556b9bf-5kzrd"] Nov 02 23:19:04.985849 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:04.983722 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-027689-5b9556b9bf-5kzrd"] Nov 02 23:19:04.985849 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:04.983851 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-027689-5b9556b9bf-5kzrd" Nov 02 23:19:04.985849 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:04.984282 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whxbj\" (UniqueName: \"kubernetes.io/projected/dd2370ae-3618-4464-939d-1821c4af19bd-kube-api-access-whxbj\") pod \"payload-pre-processing-e2e-ipp-a-027689-f647578d7-4jc4f\" (UID: \"dd2370ae-3618-4464-939d-1821c4af19bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-f647578d7-4jc4f" Nov 02 23:19:04.985849 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:04.984395 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2370ae-3618-4464-939d-1821c4af19bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-027689-f647578d7-4jc4f\" (UID: \"dd2370ae-3618-4464-939d-1821c4af19bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-f647578d7-4jc4f" Nov 02 23:19:04.985849 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:04.985341 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2370ae-3618-4464-939d-1821c4af19bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-027689-f647578d7-4jc4f\" (UID: \"dd2370ae-3618-4464-939d-1821c4af19bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-f647578d7-4jc4f" Nov 02 23:19:05.042922 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:05.042873 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whxbj\" (UniqueName: \"kubernetes.io/projected/dd2370ae-3618-4464-939d-1821c4af19bd-kube-api-access-whxbj\") pod \"payload-pre-processing-e2e-ipp-a-027689-f647578d7-4jc4f\" (UID: \"dd2370ae-3618-4464-939d-1821c4af19bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-f647578d7-4jc4f" Nov 02 23:19:05.085096 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:05.085061 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f3b39d-cd9c-46ae-86ee-83274a0d6d6c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-027689-5b9556b9bf-5kzrd\" (UID: \"77f3b39d-cd9c-46ae-86ee-83274a0d6d6c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-027689-5b9556b9bf-5kzrd" Nov 02 23:19:05.085259 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:05.085131 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wdrs\" (UniqueName: \"kubernetes.io/projected/77f3b39d-cd9c-46ae-86ee-83274a0d6d6c-kube-api-access-8wdrs\") pod \"payload-processing-e2e-ipp-a-027689-5b9556b9bf-5kzrd\" (UID: \"77f3b39d-cd9c-46ae-86ee-83274a0d6d6c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-027689-5b9556b9bf-5kzrd" Nov 02 23:19:05.186263 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:05.186232 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f3b39d-cd9c-46ae-86ee-83274a0d6d6c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-027689-5b9556b9bf-5kzrd\" (UID: \"77f3b39d-cd9c-46ae-86ee-83274a0d6d6c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-027689-5b9556b9bf-5kzrd" Nov 02 23:19:05.186459 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:05.186298 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wdrs\" (UniqueName: \"kubernetes.io/projected/77f3b39d-cd9c-46ae-86ee-83274a0d6d6c-kube-api-access-8wdrs\") pod \"payload-processing-e2e-ipp-a-027689-5b9556b9bf-5kzrd\" (UID: \"77f3b39d-cd9c-46ae-86ee-83274a0d6d6c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-027689-5b9556b9bf-5kzrd" Nov 02 23:19:05.187002 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:05.186975 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f3b39d-cd9c-46ae-86ee-83274a0d6d6c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-027689-5b9556b9bf-5kzrd\" (UID: \"77f3b39d-cd9c-46ae-86ee-83274a0d6d6c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-027689-5b9556b9bf-5kzrd" Nov 02 23:19:05.240502 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:05.240472 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wdrs\" (UniqueName: \"kubernetes.io/projected/77f3b39d-cd9c-46ae-86ee-83274a0d6d6c-kube-api-access-8wdrs\") pod \"payload-processing-e2e-ipp-a-027689-5b9556b9bf-5kzrd\" (UID: \"77f3b39d-cd9c-46ae-86ee-83274a0d6d6c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-027689-5b9556b9bf-5kzrd" Nov 02 23:19:05.299061 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:05.299030 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-f647578d7-4jc4f" Nov 02 23:19:05.304484 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:05.304465 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-027689-5b9556b9bf-5kzrd" Nov 02 23:19:05.666976 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:05.666947 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8" Nov 02 23:19:05.692306 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:05.692279 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-f647578d7-4jc4f"] Nov 02 23:19:05.692306 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:19:05.692282 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd2370ae_3618_4464_939d_1821c4af19bd.slice/crio-5ee37e24169776af0c850e4a43e1de56cd2dd964276140b177a42e3b94fb77fd WatchSource:0}: Error finding container 5ee37e24169776af0c850e4a43e1de56cd2dd964276140b177a42e3b94fb77fd: Status 404 returned error can't find the container with id 5ee37e24169776af0c850e4a43e1de56cd2dd964276140b177a42e3b94fb77fd Nov 02 23:19:05.702876 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:05.702851 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-f647578d7-4jc4f" event={"ID":"dd2370ae-3618-4464-939d-1821c4af19bd","Type":"ContainerStarted","Data":"5ee37e24169776af0c850e4a43e1de56cd2dd964276140b177a42e3b94fb77fd"} Nov 02 23:19:05.738803 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:05.738777 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-027689-5b9556b9bf-5kzrd"] Nov 02 23:19:05.740701 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:19:05.740677 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77f3b39d_cd9c_46ae_86ee_83274a0d6d6c.slice/crio-d7813f797e32c0c7cf1bd70c6376d13ff46a6d1fa972a3dff4e623e828b65e70 WatchSource:0}: Error finding container d7813f797e32c0c7cf1bd70c6376d13ff46a6d1fa972a3dff4e623e828b65e70: Status 404 returned error can't find the container with id d7813f797e32c0c7cf1bd70c6376d13ff46a6d1fa972a3dff4e623e828b65e70 Nov 02 23:19:05.821662 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:05.821628 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" Nov 02 23:19:05.823001 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:05.822971 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" Nov 02 23:19:06.711121 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:06.711075 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-027689-5b9556b9bf-5kzrd" event={"ID":"77f3b39d-cd9c-46ae-86ee-83274a0d6d6c","Type":"ContainerStarted","Data":"1a7c550bf4772935b692ef2543bf0e95dfa6a3dac843d4c1aa63f465e4c51cb3"} Nov 02 23:19:06.711588 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:06.711144 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-027689-5b9556b9bf-5kzrd" event={"ID":"77f3b39d-cd9c-46ae-86ee-83274a0d6d6c","Type":"ContainerStarted","Data":"d7813f797e32c0c7cf1bd70c6376d13ff46a6d1fa972a3dff4e623e828b65e70"} Nov 02 23:19:06.711588 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:06.711175 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-027689-5b9556b9bf-5kzrd" Nov 02 23:19:06.712665 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:06.712641 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-f647578d7-4jc4f" event={"ID":"dd2370ae-3618-4464-939d-1821c4af19bd","Type":"ContainerStarted","Data":"bd99cdae79adce5ad2c830ed4a2fec0a7115eb9ce7a51285f7ab29cfbd579b0f"} Nov 02 23:19:06.712754 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:06.712701 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-f647578d7-4jc4f" Nov 02 23:19:06.824964 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:06.824904 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-027689-5b9556b9bf-5kzrd" podStartSLOduration=2.824872652 podStartE2EDuration="2.824872652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:19:06.821478101 +0000 UTC m=+1595.790009885" watchObservedRunningTime="2026-11-02 23:19:06.824872652 +0000 UTC m=+1595.793404435" Nov 02 23:19:06.938836 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:06.938519 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-f647578d7-4jc4f" podStartSLOduration=2.9384958389999998 podStartE2EDuration="2.938495839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:19:06.929254316 +0000 UTC m=+1595.897786100" watchObservedRunningTime="2026-11-02 23:19:06.938495839 +0000 UTC m=+1595.907027617" Nov 02 23:19:09.884270 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:19:09.884237 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:19:16.770048 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:16.770011 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946"] Nov 02 23:19:16.778460 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:16.778426 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" Nov 02 23:19:16.830488 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:16.830457 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-818862-openshift-default-dockercfg-4p874\"" Nov 02 23:19:16.858961 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:16.858923 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9425db69-12a7-480d-b9a5-ef92b281a8ff-credential-socket\") pod \"e2e-api-b-818862-openshift-default-667cf76c7-v8946\" (UID: \"9425db69-12a7-480d-b9a5-ef92b281a8ff\") " pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" Nov 02 23:19:16.859130 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:16.858980 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9425db69-12a7-480d-b9a5-ef92b281a8ff-istio-envoy\") pod \"e2e-api-b-818862-openshift-default-667cf76c7-v8946\" (UID: \"9425db69-12a7-480d-b9a5-ef92b281a8ff\") " pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" Nov 02 23:19:16.859130 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:16.859110 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9425db69-12a7-480d-b9a5-ef92b281a8ff-istio-token\") pod \"e2e-api-b-818862-openshift-default-667cf76c7-v8946\" (UID: \"9425db69-12a7-480d-b9a5-ef92b281a8ff\") " pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" Nov 02 23:19:16.859230 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:16.859156 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9425db69-12a7-480d-b9a5-ef92b281a8ff-istiod-ca-cert\") pod \"e2e-api-b-818862-openshift-default-667cf76c7-v8946\" (UID: \"9425db69-12a7-480d-b9a5-ef92b281a8ff\") " pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" Nov 02 23:19:16.859316 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:16.859294 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9425db69-12a7-480d-b9a5-ef92b281a8ff-workload-socket\") pod \"e2e-api-b-818862-openshift-default-667cf76c7-v8946\" (UID: \"9425db69-12a7-480d-b9a5-ef92b281a8ff\") " pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" Nov 02 23:19:16.859355 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:16.859323 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9425db69-12a7-480d-b9a5-ef92b281a8ff-istio-podinfo\") pod \"e2e-api-b-818862-openshift-default-667cf76c7-v8946\" (UID: \"9425db69-12a7-480d-b9a5-ef92b281a8ff\") " pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" Nov 02 23:19:16.859459 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:16.859443 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9425db69-12a7-480d-b9a5-ef92b281a8ff-workload-certs\") pod \"e2e-api-b-818862-openshift-default-667cf76c7-v8946\" (UID: \"9425db69-12a7-480d-b9a5-ef92b281a8ff\") " pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" Nov 02 23:19:16.859493 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:16.859483 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9425db69-12a7-480d-b9a5-ef92b281a8ff-istio-data\") pod \"e2e-api-b-818862-openshift-default-667cf76c7-v8946\" (UID: \"9425db69-12a7-480d-b9a5-ef92b281a8ff\") " pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" Nov 02 23:19:16.859625 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:16.859611 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plrkh\" (UniqueName: \"kubernetes.io/projected/9425db69-12a7-480d-b9a5-ef92b281a8ff-kube-api-access-plrkh\") pod \"e2e-api-b-818862-openshift-default-667cf76c7-v8946\" (UID: \"9425db69-12a7-480d-b9a5-ef92b281a8ff\") " pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" Nov 02 23:19:16.881354 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:16.881323 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946"] Nov 02 23:19:16.894253 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:16.894168 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f"] Nov 02 23:19:16.894497 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:16.894449 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" podUID="ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b" containerName="istio-proxy" containerID="cri-o://49abda58f880b7c522b788b8ee508aee1136589b658e424dea48ca2cec37fdad" gracePeriod=30 Nov 02 23:19:16.918288 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:16.918254 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb"] Nov 02 23:19:16.918911 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:16.918826 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" podUID="8dd885dc-9e61-4303-a916-f422ab5eb322" containerName="istio-proxy" containerID="cri-o://5d1bfd0540626aa083b146cfb5334b059482b8fca8d4e6531a28d9fa58b01412" gracePeriod=30 Nov 02 23:19:16.928703 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:16.928677 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp"] Nov 02 23:19:16.928879 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:16.928858 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" podUID="0450c8f1-c6f6-4d65-a0a0-7e954133793d" containerName="istio-proxy" containerID="cri-o://0aef3170f0e567d4efbac8e08eb72722c8b739ce06e6d888866cd0e04475081f" gracePeriod=30 Nov 02 23:19:16.941790 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:16.941752 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk"] Nov 02 23:19:16.942007 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:16.941983 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" podUID="ca62790c-9f83-45df-9bb8-6360b271dab0" containerName="istio-proxy" containerID="cri-o://6c99bd4e46ace22a9a0c773655251c1621120c4d6925fe3892092c2378c731f1" gracePeriod=30 Nov 02 23:19:16.961325 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:16.961294 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9425db69-12a7-480d-b9a5-ef92b281a8ff-istio-token\") pod \"e2e-api-b-818862-openshift-default-667cf76c7-v8946\" (UID: \"9425db69-12a7-480d-b9a5-ef92b281a8ff\") " pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" Nov 02 23:19:16.961494 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:16.961335 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9425db69-12a7-480d-b9a5-ef92b281a8ff-istiod-ca-cert\") pod \"e2e-api-b-818862-openshift-default-667cf76c7-v8946\" (UID: \"9425db69-12a7-480d-b9a5-ef92b281a8ff\") " pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" Nov 02 23:19:16.961494 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:16.961369 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9425db69-12a7-480d-b9a5-ef92b281a8ff-workload-socket\") pod \"e2e-api-b-818862-openshift-default-667cf76c7-v8946\" (UID: \"9425db69-12a7-480d-b9a5-ef92b281a8ff\") " pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" Nov 02 23:19:16.961494 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:16.961390 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9425db69-12a7-480d-b9a5-ef92b281a8ff-istio-podinfo\") pod \"e2e-api-b-818862-openshift-default-667cf76c7-v8946\" (UID: \"9425db69-12a7-480d-b9a5-ef92b281a8ff\") " pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" Nov 02 23:19:16.961494 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:16.961461 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9425db69-12a7-480d-b9a5-ef92b281a8ff-workload-certs\") pod \"e2e-api-b-818862-openshift-default-667cf76c7-v8946\" (UID: \"9425db69-12a7-480d-b9a5-ef92b281a8ff\") " pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" Nov 02 23:19:16.961724 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:16.961494 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9425db69-12a7-480d-b9a5-ef92b281a8ff-istio-data\") pod \"e2e-api-b-818862-openshift-default-667cf76c7-v8946\" (UID: \"9425db69-12a7-480d-b9a5-ef92b281a8ff\") " pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" Nov 02 23:19:16.961724 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:16.961566 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plrkh\" (UniqueName: \"kubernetes.io/projected/9425db69-12a7-480d-b9a5-ef92b281a8ff-kube-api-access-plrkh\") pod \"e2e-api-b-818862-openshift-default-667cf76c7-v8946\" (UID: \"9425db69-12a7-480d-b9a5-ef92b281a8ff\") " pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" Nov 02 23:19:16.961724 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:16.961672 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9425db69-12a7-480d-b9a5-ef92b281a8ff-credential-socket\") pod \"e2e-api-b-818862-openshift-default-667cf76c7-v8946\" (UID: \"9425db69-12a7-480d-b9a5-ef92b281a8ff\") " pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" Nov 02 23:19:16.961724 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:16.961702 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9425db69-12a7-480d-b9a5-ef92b281a8ff-istio-envoy\") pod \"e2e-api-b-818862-openshift-default-667cf76c7-v8946\" (UID: \"9425db69-12a7-480d-b9a5-ef92b281a8ff\") " pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" Nov 02 23:19:16.962118 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:16.962086 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9425db69-12a7-480d-b9a5-ef92b281a8ff-workload-socket\") pod \"e2e-api-b-818862-openshift-default-667cf76c7-v8946\" (UID: \"9425db69-12a7-480d-b9a5-ef92b281a8ff\") " pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" Nov 02 23:19:16.962558 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:16.962535 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9425db69-12a7-480d-b9a5-ef92b281a8ff-workload-certs\") pod \"e2e-api-b-818862-openshift-default-667cf76c7-v8946\" (UID: \"9425db69-12a7-480d-b9a5-ef92b281a8ff\") " pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" Nov 02 23:19:16.962803 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:16.962782 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9425db69-12a7-480d-b9a5-ef92b281a8ff-istio-data\") pod \"e2e-api-b-818862-openshift-default-667cf76c7-v8946\" (UID: \"9425db69-12a7-480d-b9a5-ef92b281a8ff\") " pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" Nov 02 23:19:16.963092 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:16.963069 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9425db69-12a7-480d-b9a5-ef92b281a8ff-credential-socket\") pod \"e2e-api-b-818862-openshift-default-667cf76c7-v8946\" (UID: \"9425db69-12a7-480d-b9a5-ef92b281a8ff\") " pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" Nov 02 23:19:16.963502 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:16.963470 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9425db69-12a7-480d-b9a5-ef92b281a8ff-istiod-ca-cert\") pod \"e2e-api-b-818862-openshift-default-667cf76c7-v8946\" (UID: \"9425db69-12a7-480d-b9a5-ef92b281a8ff\") " pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" Nov 02 23:19:16.964364 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:16.964340 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9425db69-12a7-480d-b9a5-ef92b281a8ff-istio-podinfo\") pod \"e2e-api-b-818862-openshift-default-667cf76c7-v8946\" (UID: \"9425db69-12a7-480d-b9a5-ef92b281a8ff\") " pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" Nov 02 23:19:16.965057 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:16.965040 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9425db69-12a7-480d-b9a5-ef92b281a8ff-istio-envoy\") pod \"e2e-api-b-818862-openshift-default-667cf76c7-v8946\" (UID: \"9425db69-12a7-480d-b9a5-ef92b281a8ff\") " pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" Nov 02 23:19:17.067960 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:17.067593 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9425db69-12a7-480d-b9a5-ef92b281a8ff-istio-token\") pod \"e2e-api-b-818862-openshift-default-667cf76c7-v8946\" (UID: \"9425db69-12a7-480d-b9a5-ef92b281a8ff\") " pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" Nov 02 23:19:17.104070 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:17.099178 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plrkh\" (UniqueName: \"kubernetes.io/projected/9425db69-12a7-480d-b9a5-ef92b281a8ff-kube-api-access-plrkh\") pod \"e2e-api-b-818862-openshift-default-667cf76c7-v8946\" (UID: \"9425db69-12a7-480d-b9a5-ef92b281a8ff\") " pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" Nov 02 23:19:17.112918 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:17.110936 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" Nov 02 23:19:17.633305 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:17.633278 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946"] Nov 02 23:19:17.633732 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:19:17.633707 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9425db69_12a7_480d_b9a5_ef92b281a8ff.slice/crio-6d29fff5b720f795a40ffe84fb0f6239fe3c40d79a367229718d47079f600f1c WatchSource:0}: Error finding container 6d29fff5b720f795a40ffe84fb0f6239fe3c40d79a367229718d47079f600f1c: Status 404 returned error can't find the container with id 6d29fff5b720f795a40ffe84fb0f6239fe3c40d79a367229718d47079f600f1c Nov 02 23:19:17.636248 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:17.636212 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:19:17.636359 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:17.636295 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:19:17.636359 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:17.636340 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:19:17.722832 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:17.722809 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-f647578d7-4jc4f" Nov 02 23:19:17.722978 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:17.722878 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-027689-5b9556b9bf-5kzrd" Nov 02 23:19:17.794280 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:17.794236 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" event={"ID":"9425db69-12a7-480d-b9a5-ef92b281a8ff","Type":"ContainerStarted","Data":"701de5b7736526c41ee427ab87c782202f12e81d4468487a14a205faa9b5ebe0"} Nov 02 23:19:17.794754 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:17.794298 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" event={"ID":"9425db69-12a7-480d-b9a5-ef92b281a8ff","Type":"ContainerStarted","Data":"6d29fff5b720f795a40ffe84fb0f6239fe3c40d79a367229718d47079f600f1c"} Nov 02 23:19:18.052466 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:18.052411 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" podStartSLOduration=2.052397198 podStartE2EDuration="2.052397198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:19:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:19:18.051079245 +0000 UTC m=+1607.019611030" watchObservedRunningTime="2026-11-02 23:19:18.052397198 +0000 UTC m=+1607.020928981" Nov 02 23:19:18.111390 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:18.111352 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" Nov 02 23:19:18.117120 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:18.117094 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" Nov 02 23:19:18.692733 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:18.692696 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw"] Nov 02 23:19:18.693037 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:18.692998 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" podUID="5662cc68-8f5b-48f8-8ed9-858e5595524d" containerName="istio-proxy" containerID="cri-o://240f24a25f4ee2b62df78be0ac271323c6968f322aba72c0f11082bd116980bc" gracePeriod=30 Nov 02 23:19:18.800764 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:18.800733 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" Nov 02 23:19:18.801717 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:18.801700 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" Nov 02 23:19:19.039866 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:19.039786 5076 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:19:19.039866 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:19.039843 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" podUID="8dd885dc-9e61-4303-a916-f422ab5eb322" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:19:21.598623 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:21.598587 5076 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:19:21.599145 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:21.598652 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" podUID="ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:19:22.522981 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:19:22.522395 5076 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca62790c_9f83_45df_9bb8_6360b271dab0.slice/crio-conmon-6c99bd4e46ace22a9a0c773655251c1621120c4d6925fe3892092c2378c731f1.scope\": RecentStats: unable to find data in memory cache]" Nov 02 23:19:22.674526 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.674494 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" Nov 02 23:19:22.773986 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.773920 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" Nov 02 23:19:22.792607 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.792560 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" Nov 02 23:19:22.801122 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.801101 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" Nov 02 23:19:22.836982 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.836941 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-kube-api-access-j2pvj\" (UniqueName: \"kubernetes.io/projected/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-kube-api-access-j2pvj\") pod \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\" (UID: \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\") " Nov 02 23:19:22.837156 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.836985 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-credential-socket\") pod \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\" (UID: \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\") " Nov 02 23:19:22.837156 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.837041 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-istio-podinfo\") pod \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\" (UID: \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\") " Nov 02 23:19:22.837156 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.837119 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-workload-certs\") pod \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\" (UID: \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\") " Nov 02 23:19:22.837156 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.837148 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-istio-data\") pod \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\" (UID: \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\") " Nov 02 23:19:22.837374 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.837176 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-istiod-ca-cert\") pod \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\" (UID: \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\") " Nov 02 23:19:22.837374 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.837276 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-workload-socket\") pod \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\" (UID: \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\") " Nov 02 23:19:22.837374 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.837316 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-istio-envoy\") pod \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\" (UID: \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\") " Nov 02 23:19:22.837519 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.837498 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-istio-token\" (UniqueName: \"kubernetes.io/projected/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-istio-token\") pod \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\" (UID: \"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b\") " Nov 02 23:19:22.838236 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.838208 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-workload-certs" pod "ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b" (UID: "ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:22.838485 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.838450 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-istio-data" pod "ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b" (UID: "ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:22.838485 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.838472 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-istiod-ca-cert" pod "ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b" (UID: "ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:19:22.838701 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.838680 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-workload-socket" pod "ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b" (UID: "ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:22.838820 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.838795 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-credential-socket" pod "ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b" (UID: "ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:22.841276 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.841247 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-kube-api-access-j2pvj" pod "ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b" (UID: "ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b"). InnerVolumeSpecName "kube-api-access-j2pvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:19:22.841417 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.841320 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-istio-podinfo" pod "ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b" (UID: "ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:19:22.841417 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.841338 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-istio-token" pod "ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b" (UID: "ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:19:22.842634 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.842611 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-istio-envoy" pod "ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b" (UID: "ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:22.857971 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.857926 5076 generic.go:356] "Generic (PLEG): container finished" podID="ca62790c-9f83-45df-9bb8-6360b271dab0" containerID="6c99bd4e46ace22a9a0c773655251c1621120c4d6925fe3892092c2378c731f1" exitCode=0 Nov 02 23:19:22.858167 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.858014 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" Nov 02 23:19:22.858167 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.858038 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" event={"ID":"ca62790c-9f83-45df-9bb8-6360b271dab0","Type":"ContainerDied","Data":"6c99bd4e46ace22a9a0c773655251c1621120c4d6925fe3892092c2378c731f1"} Nov 02 23:19:22.858167 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.858078 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk" event={"ID":"ca62790c-9f83-45df-9bb8-6360b271dab0","Type":"ContainerDied","Data":"7a28035e8eb7f2df9854bf06026b9df25e9da4f66b4914a9489410199048f8db"} Nov 02 23:19:22.858167 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.858098 5076 scope.go:122] "RemoveContainer" containerID="6c99bd4e46ace22a9a0c773655251c1621120c4d6925fe3892092c2378c731f1" Nov 02 23:19:22.859803 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.859773 5076 generic.go:356] "Generic (PLEG): container finished" podID="ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b" containerID="49abda58f880b7c522b788b8ee508aee1136589b658e424dea48ca2cec37fdad" exitCode=0 Nov 02 23:19:22.859959 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.859818 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" Nov 02 23:19:22.859959 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.859844 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" event={"ID":"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b","Type":"ContainerDied","Data":"49abda58f880b7c522b788b8ee508aee1136589b658e424dea48ca2cec37fdad"} Nov 02 23:19:22.859959 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.859913 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f" event={"ID":"ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b","Type":"ContainerDied","Data":"eb55c978a06dd62e8fdfdd61a1a86657ea2cd382ff7ca5a1eedd268626e04f9d"} Nov 02 23:19:22.861948 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.861921 5076 generic.go:356] "Generic (PLEG): container finished" podID="8dd885dc-9e61-4303-a916-f422ab5eb322" containerID="5d1bfd0540626aa083b146cfb5334b059482b8fca8d4e6531a28d9fa58b01412" exitCode=0 Nov 02 23:19:22.862212 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.862053 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" event={"ID":"8dd885dc-9e61-4303-a916-f422ab5eb322","Type":"ContainerDied","Data":"5d1bfd0540626aa083b146cfb5334b059482b8fca8d4e6531a28d9fa58b01412"} Nov 02 23:19:22.862212 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.862064 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" Nov 02 23:19:22.862212 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.862101 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb" event={"ID":"8dd885dc-9e61-4303-a916-f422ab5eb322","Type":"ContainerDied","Data":"5f9d64653bf1b476650b136748f1ced0d88a4865be42ed1f15455a2be7bde197"} Nov 02 23:19:22.863807 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.863783 5076 generic.go:356] "Generic (PLEG): container finished" podID="0450c8f1-c6f6-4d65-a0a0-7e954133793d" containerID="0aef3170f0e567d4efbac8e08eb72722c8b739ce06e6d888866cd0e04475081f" exitCode=0 Nov 02 23:19:22.863919 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.863826 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" Nov 02 23:19:22.863919 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.863852 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" event={"ID":"0450c8f1-c6f6-4d65-a0a0-7e954133793d","Type":"ContainerDied","Data":"0aef3170f0e567d4efbac8e08eb72722c8b739ce06e6d888866cd0e04475081f"} Nov 02 23:19:22.863919 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.863918 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp" event={"ID":"0450c8f1-c6f6-4d65-a0a0-7e954133793d","Type":"ContainerDied","Data":"05fe7095208864455dde002b63539781dacfb4e1b47ef90ec4cc50eb011b0386"} Nov 02 23:19:22.870477 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.870362 5076 scope.go:122] "RemoveContainer" containerID="6c99bd4e46ace22a9a0c773655251c1621120c4d6925fe3892092c2378c731f1" Nov 02 23:19:22.870722 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:19:22.870694 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c99bd4e46ace22a9a0c773655251c1621120c4d6925fe3892092c2378c731f1\": container with ID starting with 6c99bd4e46ace22a9a0c773655251c1621120c4d6925fe3892092c2378c731f1 not found: ID does not exist" containerID="6c99bd4e46ace22a9a0c773655251c1621120c4d6925fe3892092c2378c731f1" Nov 02 23:19:22.870796 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.870754 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c99bd4e46ace22a9a0c773655251c1621120c4d6925fe3892092c2378c731f1"} err="failed to get container status \"6c99bd4e46ace22a9a0c773655251c1621120c4d6925fe3892092c2378c731f1\": rpc error: code = NotFound desc = could not find container \"6c99bd4e46ace22a9a0c773655251c1621120c4d6925fe3892092c2378c731f1\": container with ID starting with 6c99bd4e46ace22a9a0c773655251c1621120c4d6925fe3892092c2378c731f1 not found: ID does not exist" Nov 02 23:19:22.870796 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.870779 5076 scope.go:122] "RemoveContainer" containerID="49abda58f880b7c522b788b8ee508aee1136589b658e424dea48ca2cec37fdad" Nov 02 23:19:22.881828 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.881807 5076 scope.go:122] "RemoveContainer" containerID="49abda58f880b7c522b788b8ee508aee1136589b658e424dea48ca2cec37fdad" Nov 02 23:19:22.882140 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:19:22.882113 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49abda58f880b7c522b788b8ee508aee1136589b658e424dea48ca2cec37fdad\": container with ID starting with 49abda58f880b7c522b788b8ee508aee1136589b658e424dea48ca2cec37fdad not found: ID does not exist" containerID="49abda58f880b7c522b788b8ee508aee1136589b658e424dea48ca2cec37fdad" Nov 02 23:19:22.882317 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.882149 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49abda58f880b7c522b788b8ee508aee1136589b658e424dea48ca2cec37fdad"} err="failed to get container status \"49abda58f880b7c522b788b8ee508aee1136589b658e424dea48ca2cec37fdad\": rpc error: code = NotFound desc = could not find container \"49abda58f880b7c522b788b8ee508aee1136589b658e424dea48ca2cec37fdad\": container with ID starting with 49abda58f880b7c522b788b8ee508aee1136589b658e424dea48ca2cec37fdad not found: ID does not exist" Nov 02 23:19:22.882317 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.882176 5076 scope.go:122] "RemoveContainer" containerID="5d1bfd0540626aa083b146cfb5334b059482b8fca8d4e6531a28d9fa58b01412" Nov 02 23:19:22.901150 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.901115 5076 scope.go:122] "RemoveContainer" containerID="5d1bfd0540626aa083b146cfb5334b059482b8fca8d4e6531a28d9fa58b01412" Nov 02 23:19:22.901561 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:19:22.901523 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d1bfd0540626aa083b146cfb5334b059482b8fca8d4e6531a28d9fa58b01412\": container with ID starting with 5d1bfd0540626aa083b146cfb5334b059482b8fca8d4e6531a28d9fa58b01412 not found: ID does not exist" containerID="5d1bfd0540626aa083b146cfb5334b059482b8fca8d4e6531a28d9fa58b01412" Nov 02 23:19:22.901667 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.901573 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d1bfd0540626aa083b146cfb5334b059482b8fca8d4e6531a28d9fa58b01412"} err="failed to get container status \"5d1bfd0540626aa083b146cfb5334b059482b8fca8d4e6531a28d9fa58b01412\": rpc error: code = NotFound desc = could not find container \"5d1bfd0540626aa083b146cfb5334b059482b8fca8d4e6531a28d9fa58b01412\": container with ID starting with 5d1bfd0540626aa083b146cfb5334b059482b8fca8d4e6531a28d9fa58b01412 not found: ID does not exist" Nov 02 23:19:22.901667 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.901602 5076 scope.go:122] "RemoveContainer" containerID="0aef3170f0e567d4efbac8e08eb72722c8b739ce06e6d888866cd0e04475081f" Nov 02 23:19:22.911338 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.911309 5076 scope.go:122] "RemoveContainer" containerID="0aef3170f0e567d4efbac8e08eb72722c8b739ce06e6d888866cd0e04475081f" Nov 02 23:19:22.911688 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:19:22.911659 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aef3170f0e567d4efbac8e08eb72722c8b739ce06e6d888866cd0e04475081f\": container with ID starting with 0aef3170f0e567d4efbac8e08eb72722c8b739ce06e6d888866cd0e04475081f not found: ID does not exist" containerID="0aef3170f0e567d4efbac8e08eb72722c8b739ce06e6d888866cd0e04475081f" Nov 02 23:19:22.911797 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.911703 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aef3170f0e567d4efbac8e08eb72722c8b739ce06e6d888866cd0e04475081f"} err="failed to get container status \"0aef3170f0e567d4efbac8e08eb72722c8b739ce06e6d888866cd0e04475081f\": rpc error: code = NotFound desc = could not find container \"0aef3170f0e567d4efbac8e08eb72722c8b739ce06e6d888866cd0e04475081f\": container with ID starting with 0aef3170f0e567d4efbac8e08eb72722c8b739ce06e6d888866cd0e04475081f not found: ID does not exist" Nov 02 23:19:22.938527 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.938501 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca62790c-9f83-45df-9bb8-6360b271dab0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca62790c-9f83-45df-9bb8-6360b271dab0-istio-envoy\") pod \"ca62790c-9f83-45df-9bb8-6360b271dab0\" (UID: \"ca62790c-9f83-45df-9bb8-6360b271dab0\") " Nov 02 23:19:22.938634 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.938533 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca62790c-9f83-45df-9bb8-6360b271dab0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca62790c-9f83-45df-9bb8-6360b271dab0-istio-data\") pod \"ca62790c-9f83-45df-9bb8-6360b271dab0\" (UID: \"ca62790c-9f83-45df-9bb8-6360b271dab0\") " Nov 02 23:19:22.938634 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.938552 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dd885dc-9e61-4303-a916-f422ab5eb322-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dd885dc-9e61-4303-a916-f422ab5eb322-istio-data\") pod \"8dd885dc-9e61-4303-a916-f422ab5eb322\" (UID: \"8dd885dc-9e61-4303-a916-f422ab5eb322\") " Nov 02 23:19:22.938634 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.938586 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dd885dc-9e61-4303-a916-f422ab5eb322-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd885dc-9e61-4303-a916-f422ab5eb322-workload-socket\") pod \"8dd885dc-9e61-4303-a916-f422ab5eb322\" (UID: \"8dd885dc-9e61-4303-a916-f422ab5eb322\") " Nov 02 23:19:22.938634 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.938615 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dd885dc-9e61-4303-a916-f422ab5eb322-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dd885dc-9e61-4303-a916-f422ab5eb322-istiod-ca-cert\") pod \"8dd885dc-9e61-4303-a916-f422ab5eb322\" (UID: \"8dd885dc-9e61-4303-a916-f422ab5eb322\") " Nov 02 23:19:22.938854 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.938635 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0450c8f1-c6f6-4d65-a0a0-7e954133793d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0450c8f1-c6f6-4d65-a0a0-7e954133793d-istio-envoy\") pod \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\" (UID: \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\") " Nov 02 23:19:22.938854 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.938655 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0450c8f1-c6f6-4d65-a0a0-7e954133793d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0450c8f1-c6f6-4d65-a0a0-7e954133793d-credential-socket\") pod \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\" (UID: \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\") " Nov 02 23:19:22.938854 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.938695 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca62790c-9f83-45df-9bb8-6360b271dab0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca62790c-9f83-45df-9bb8-6360b271dab0-istio-podinfo\") pod \"ca62790c-9f83-45df-9bb8-6360b271dab0\" (UID: \"ca62790c-9f83-45df-9bb8-6360b271dab0\") " Nov 02 23:19:22.938854 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.938733 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dd885dc-9e61-4303-a916-f422ab5eb322-kube-api-access-dkpkq\" (UniqueName: \"kubernetes.io/projected/8dd885dc-9e61-4303-a916-f422ab5eb322-kube-api-access-dkpkq\") pod \"8dd885dc-9e61-4303-a916-f422ab5eb322\" (UID: \"8dd885dc-9e61-4303-a916-f422ab5eb322\") " Nov 02 23:19:22.938854 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.938762 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0450c8f1-c6f6-4d65-a0a0-7e954133793d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0450c8f1-c6f6-4d65-a0a0-7e954133793d-istiod-ca-cert\") pod \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\" (UID: \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\") " Nov 02 23:19:22.938854 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.938797 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca62790c-9f83-45df-9bb8-6360b271dab0-istio-data" pod "ca62790c-9f83-45df-9bb8-6360b271dab0" (UID: "ca62790c-9f83-45df-9bb8-6360b271dab0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:22.938854 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.938829 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0450c8f1-c6f6-4d65-a0a0-7e954133793d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0450c8f1-c6f6-4d65-a0a0-7e954133793d-workload-certs\") pod \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\" (UID: \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\") " Nov 02 23:19:22.939224 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.938831 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd885dc-9e61-4303-a916-f422ab5eb322-workload-socket" pod "8dd885dc-9e61-4303-a916-f422ab5eb322" (UID: "8dd885dc-9e61-4303-a916-f422ab5eb322"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:22.939224 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.938878 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dd885dc-9e61-4303-a916-f422ab5eb322-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dd885dc-9e61-4303-a916-f422ab5eb322-workload-certs\") pod \"8dd885dc-9e61-4303-a916-f422ab5eb322\" (UID: \"8dd885dc-9e61-4303-a916-f422ab5eb322\") " Nov 02 23:19:22.939224 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.938933 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd885dc-9e61-4303-a916-f422ab5eb322-istio-data" pod "8dd885dc-9e61-4303-a916-f422ab5eb322" (UID: "8dd885dc-9e61-4303-a916-f422ab5eb322"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:22.939224 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.938952 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0450c8f1-c6f6-4d65-a0a0-7e954133793d-credential-socket" pod "0450c8f1-c6f6-4d65-a0a0-7e954133793d" (UID: "0450c8f1-c6f6-4d65-a0a0-7e954133793d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:22.939224 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.938957 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0450c8f1-c6f6-4d65-a0a0-7e954133793d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0450c8f1-c6f6-4d65-a0a0-7e954133793d-istio-podinfo\") pod \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\" (UID: \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\") " Nov 02 23:19:22.939224 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.939023 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca62790c-9f83-45df-9bb8-6360b271dab0-kube-api-access-8ln22\" (UniqueName: \"kubernetes.io/projected/ca62790c-9f83-45df-9bb8-6360b271dab0-kube-api-access-8ln22\") pod \"ca62790c-9f83-45df-9bb8-6360b271dab0\" (UID: \"ca62790c-9f83-45df-9bb8-6360b271dab0\") " Nov 02 23:19:22.939224 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.939067 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0450c8f1-c6f6-4d65-a0a0-7e954133793d-kube-api-access-zkprc\" (UniqueName: \"kubernetes.io/projected/0450c8f1-c6f6-4d65-a0a0-7e954133793d-kube-api-access-zkprc\") pod \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\" (UID: \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\") " Nov 02 23:19:22.939224 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.939097 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca62790c-9f83-45df-9bb8-6360b271dab0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca62790c-9f83-45df-9bb8-6360b271dab0-workload-socket\") pod \"ca62790c-9f83-45df-9bb8-6360b271dab0\" (UID: \"ca62790c-9f83-45df-9bb8-6360b271dab0\") " Nov 02 23:19:22.939224 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.939151 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca62790c-9f83-45df-9bb8-6360b271dab0-istio-token\" (UniqueName: \"kubernetes.io/projected/ca62790c-9f83-45df-9bb8-6360b271dab0-istio-token\") pod \"ca62790c-9f83-45df-9bb8-6360b271dab0\" (UID: \"ca62790c-9f83-45df-9bb8-6360b271dab0\") " Nov 02 23:19:22.939224 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.939189 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca62790c-9f83-45df-9bb8-6360b271dab0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca62790c-9f83-45df-9bb8-6360b271dab0-istiod-ca-cert\") pod \"ca62790c-9f83-45df-9bb8-6360b271dab0\" (UID: \"ca62790c-9f83-45df-9bb8-6360b271dab0\") " Nov 02 23:19:22.939224 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.939200 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0450c8f1-c6f6-4d65-a0a0-7e954133793d-workload-certs" pod "0450c8f1-c6f6-4d65-a0a0-7e954133793d" (UID: "0450c8f1-c6f6-4d65-a0a0-7e954133793d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:22.939224 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.939118 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dd885dc-9e61-4303-a916-f422ab5eb322-istiod-ca-cert" pod "8dd885dc-9e61-4303-a916-f422ab5eb322" (UID: "8dd885dc-9e61-4303-a916-f422ab5eb322"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:19:22.939876 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.939221 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0450c8f1-c6f6-4d65-a0a0-7e954133793d-istio-token\" (UniqueName: \"kubernetes.io/projected/0450c8f1-c6f6-4d65-a0a0-7e954133793d-istio-token\") pod \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\" (UID: \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\") " Nov 02 23:19:22.939876 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.939286 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca62790c-9f83-45df-9bb8-6360b271dab0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca62790c-9f83-45df-9bb8-6360b271dab0-workload-certs\") pod \"ca62790c-9f83-45df-9bb8-6360b271dab0\" (UID: \"ca62790c-9f83-45df-9bb8-6360b271dab0\") " Nov 02 23:19:22.939876 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.939369 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dd885dc-9e61-4303-a916-f422ab5eb322-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd885dc-9e61-4303-a916-f422ab5eb322-credential-socket\") pod \"8dd885dc-9e61-4303-a916-f422ab5eb322\" (UID: \"8dd885dc-9e61-4303-a916-f422ab5eb322\") " Nov 02 23:19:22.939876 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.939425 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0450c8f1-c6f6-4d65-a0a0-7e954133793d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0450c8f1-c6f6-4d65-a0a0-7e954133793d-istio-data\") pod \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\" (UID: \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\") " Nov 02 23:19:22.939876 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.939455 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca62790c-9f83-45df-9bb8-6360b271dab0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca62790c-9f83-45df-9bb8-6360b271dab0-credential-socket\") pod \"ca62790c-9f83-45df-9bb8-6360b271dab0\" (UID: \"ca62790c-9f83-45df-9bb8-6360b271dab0\") " Nov 02 23:19:22.939876 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.939493 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0450c8f1-c6f6-4d65-a0a0-7e954133793d-istiod-ca-cert" pod "0450c8f1-c6f6-4d65-a0a0-7e954133793d" (UID: "0450c8f1-c6f6-4d65-a0a0-7e954133793d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:19:22.939876 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.939531 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dd885dc-9e61-4303-a916-f422ab5eb322-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dd885dc-9e61-4303-a916-f422ab5eb322-istio-envoy\") pod \"8dd885dc-9e61-4303-a916-f422ab5eb322\" (UID: \"8dd885dc-9e61-4303-a916-f422ab5eb322\") " Nov 02 23:19:22.939876 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.939579 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0450c8f1-c6f6-4d65-a0a0-7e954133793d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0450c8f1-c6f6-4d65-a0a0-7e954133793d-workload-socket\") pod \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\" (UID: \"0450c8f1-c6f6-4d65-a0a0-7e954133793d\") " Nov 02 23:19:22.939876 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.939628 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dd885dc-9e61-4303-a916-f422ab5eb322-istio-token\" (UniqueName: \"kubernetes.io/projected/8dd885dc-9e61-4303-a916-f422ab5eb322-istio-token\") pod \"8dd885dc-9e61-4303-a916-f422ab5eb322\" (UID: \"8dd885dc-9e61-4303-a916-f422ab5eb322\") " Nov 02 23:19:22.939876 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.939702 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8dd885dc-9e61-4303-a916-f422ab5eb322-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dd885dc-9e61-4303-a916-f422ab5eb322-istio-podinfo\") pod \"8dd885dc-9e61-4303-a916-f422ab5eb322\" (UID: \"8dd885dc-9e61-4303-a916-f422ab5eb322\") " Nov 02 23:19:22.939876 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.939774 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd885dc-9e61-4303-a916-f422ab5eb322-credential-socket" pod "8dd885dc-9e61-4303-a916-f422ab5eb322" (UID: "8dd885dc-9e61-4303-a916-f422ab5eb322"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:22.940544 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.939971 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca62790c-9f83-45df-9bb8-6360b271dab0-workload-certs" pod "ca62790c-9f83-45df-9bb8-6360b271dab0" (UID: "ca62790c-9f83-45df-9bb8-6360b271dab0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:22.940544 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.940139 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd885dc-9e61-4303-a916-f422ab5eb322-workload-certs" pod "8dd885dc-9e61-4303-a916-f422ab5eb322" (UID: "8dd885dc-9e61-4303-a916-f422ab5eb322"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:22.940544 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.940467 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:22.940544 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.940493 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca62790c-9f83-45df-9bb8-6360b271dab0-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:22.940544 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.940510 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:22.940544 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.940525 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dd885dc-9e61-4303-a916-f422ab5eb322-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:22.940544 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.940539 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:22.940984 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.940555 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:22.940984 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.940570 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd885dc-9e61-4303-a916-f422ab5eb322-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:22.940984 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.940585 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dd885dc-9e61-4303-a916-f422ab5eb322-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:22.940984 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.940599 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0450c8f1-c6f6-4d65-a0a0-7e954133793d-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:22.940984 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.940613 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:22.940984 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.940628 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0450c8f1-c6f6-4d65-a0a0-7e954133793d-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:22.940984 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.940642 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:22.940984 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.940657 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0450c8f1-c6f6-4d65-a0a0-7e954133793d-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:22.940984 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.940670 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dd885dc-9e61-4303-a916-f422ab5eb322-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:22.940984 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.940685 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca62790c-9f83-45df-9bb8-6360b271dab0-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:22.940984 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.940726 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:22.940984 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.940741 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd885dc-9e61-4303-a916-f422ab5eb322-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:22.940984 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.940757 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2pvj\" (UniqueName: \"kubernetes.io/projected/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-kube-api-access-j2pvj\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:22.940984 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.940774 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:22.941723 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.941051 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0450c8f1-c6f6-4d65-a0a0-7e954133793d-workload-socket" pod "0450c8f1-c6f6-4d65-a0a0-7e954133793d" (UID: "0450c8f1-c6f6-4d65-a0a0-7e954133793d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:22.941723 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.941241 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca62790c-9f83-45df-9bb8-6360b271dab0-istio-envoy" pod "ca62790c-9f83-45df-9bb8-6360b271dab0" (UID: "ca62790c-9f83-45df-9bb8-6360b271dab0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:22.941723 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.941298 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0450c8f1-c6f6-4d65-a0a0-7e954133793d-istio-data" pod "0450c8f1-c6f6-4d65-a0a0-7e954133793d" (UID: "0450c8f1-c6f6-4d65-a0a0-7e954133793d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:22.941723 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.941504 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca62790c-9f83-45df-9bb8-6360b271dab0-workload-socket" pod "ca62790c-9f83-45df-9bb8-6360b271dab0" (UID: "ca62790c-9f83-45df-9bb8-6360b271dab0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:22.941723 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.941518 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca62790c-9f83-45df-9bb8-6360b271dab0-credential-socket" pod "ca62790c-9f83-45df-9bb8-6360b271dab0" (UID: "ca62790c-9f83-45df-9bb8-6360b271dab0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:22.942018 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.941883 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0450c8f1-c6f6-4d65-a0a0-7e954133793d-istio-podinfo" pod "0450c8f1-c6f6-4d65-a0a0-7e954133793d" (UID: "0450c8f1-c6f6-4d65-a0a0-7e954133793d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:19:22.942018 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.941924 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca62790c-9f83-45df-9bb8-6360b271dab0-istiod-ca-cert" pod "ca62790c-9f83-45df-9bb8-6360b271dab0" (UID: "ca62790c-9f83-45df-9bb8-6360b271dab0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:19:22.942018 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.941797 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca62790c-9f83-45df-9bb8-6360b271dab0-istio-podinfo" pod "ca62790c-9f83-45df-9bb8-6360b271dab0" (UID: "ca62790c-9f83-45df-9bb8-6360b271dab0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:19:22.942492 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.942456 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0450c8f1-c6f6-4d65-a0a0-7e954133793d-istio-token" pod "0450c8f1-c6f6-4d65-a0a0-7e954133793d" (UID: "0450c8f1-c6f6-4d65-a0a0-7e954133793d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:19:22.942601 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.942499 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dd885dc-9e61-4303-a916-f422ab5eb322-kube-api-access-dkpkq" pod "8dd885dc-9e61-4303-a916-f422ab5eb322" (UID: "8dd885dc-9e61-4303-a916-f422ab5eb322"). InnerVolumeSpecName "kube-api-access-dkpkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:19:22.942666 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.942596 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd885dc-9e61-4303-a916-f422ab5eb322-istio-envoy" pod "8dd885dc-9e61-4303-a916-f422ab5eb322" (UID: "8dd885dc-9e61-4303-a916-f422ab5eb322"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:22.943446 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.943421 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dd885dc-9e61-4303-a916-f422ab5eb322-istio-token" pod "8dd885dc-9e61-4303-a916-f422ab5eb322" (UID: "8dd885dc-9e61-4303-a916-f422ab5eb322"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:19:22.943839 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.943822 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0450c8f1-c6f6-4d65-a0a0-7e954133793d-kube-api-access-zkprc" pod "0450c8f1-c6f6-4d65-a0a0-7e954133793d" (UID: "0450c8f1-c6f6-4d65-a0a0-7e954133793d"). InnerVolumeSpecName "kube-api-access-zkprc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:19:22.944008 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.943990 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0450c8f1-c6f6-4d65-a0a0-7e954133793d-istio-envoy" pod "0450c8f1-c6f6-4d65-a0a0-7e954133793d" (UID: "0450c8f1-c6f6-4d65-a0a0-7e954133793d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:22.944139 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.944117 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca62790c-9f83-45df-9bb8-6360b271dab0-istio-token" pod "ca62790c-9f83-45df-9bb8-6360b271dab0" (UID: "ca62790c-9f83-45df-9bb8-6360b271dab0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:19:22.944206 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.944124 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8dd885dc-9e61-4303-a916-f422ab5eb322-istio-podinfo" pod "8dd885dc-9e61-4303-a916-f422ab5eb322" (UID: "8dd885dc-9e61-4303-a916-f422ab5eb322"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:19:22.944258 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:22.944235 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca62790c-9f83-45df-9bb8-6360b271dab0-kube-api-access-8ln22" pod "ca62790c-9f83-45df-9bb8-6360b271dab0" (UID: "ca62790c-9f83-45df-9bb8-6360b271dab0"). InnerVolumeSpecName "kube-api-access-8ln22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:19:23.042329 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:23.042249 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0450c8f1-c6f6-4d65-a0a0-7e954133793d-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:23.042329 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:23.042280 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca62790c-9f83-45df-9bb8-6360b271dab0-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:23.042329 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:23.042289 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dd885dc-9e61-4303-a916-f422ab5eb322-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:23.042329 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:23.042298 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0450c8f1-c6f6-4d65-a0a0-7e954133793d-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:23.042329 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:23.042307 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dd885dc-9e61-4303-a916-f422ab5eb322-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:23.042329 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:23.042316 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dd885dc-9e61-4303-a916-f422ab5eb322-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:23.042329 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:23.042324 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca62790c-9f83-45df-9bb8-6360b271dab0-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:23.042329 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:23.042332 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0450c8f1-c6f6-4d65-a0a0-7e954133793d-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:23.042671 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:23.042342 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca62790c-9f83-45df-9bb8-6360b271dab0-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:23.042671 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:23.042350 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkpkq\" (UniqueName: \"kubernetes.io/projected/8dd885dc-9e61-4303-a916-f422ab5eb322-kube-api-access-dkpkq\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:23.042671 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:23.042359 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0450c8f1-c6f6-4d65-a0a0-7e954133793d-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:23.042671 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:23.042367 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ln22\" (UniqueName: \"kubernetes.io/projected/ca62790c-9f83-45df-9bb8-6360b271dab0-kube-api-access-8ln22\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:23.042671 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:23.042375 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkprc\" (UniqueName: \"kubernetes.io/projected/0450c8f1-c6f6-4d65-a0a0-7e954133793d-kube-api-access-zkprc\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:23.042671 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:23.042384 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca62790c-9f83-45df-9bb8-6360b271dab0-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:23.042671 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:23.042392 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca62790c-9f83-45df-9bb8-6360b271dab0-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:23.042671 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:23.042400 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca62790c-9f83-45df-9bb8-6360b271dab0-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:23.042671 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:23.042410 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0450c8f1-c6f6-4d65-a0a0-7e954133793d-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:23.301335 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:23.301195 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp"] Nov 02 23:19:23.304944 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:23.304915 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-csnjp"] Nov 02 23:19:23.405662 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:23.405629 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f"] Nov 02 23:19:23.410617 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:23.410591 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-88j6f"] Nov 02 23:19:23.514385 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:23.514348 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb"] Nov 02 23:19:23.518278 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:23.518252 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-65mjb"] Nov 02 23:19:23.622499 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:23.622408 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk"] Nov 02 23:19:23.625933 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:23.625903 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-qcqxk"] Nov 02 23:19:23.889217 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:23.888703 5076 generic.go:356] "Generic (PLEG): container finished" podID="5662cc68-8f5b-48f8-8ed9-858e5595524d" containerID="240f24a25f4ee2b62df78be0ac271323c6968f322aba72c0f11082bd116980bc" exitCode=0 Nov 02 23:19:23.889217 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:23.888731 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" event={"ID":"5662cc68-8f5b-48f8-8ed9-858e5595524d","Type":"ContainerDied","Data":"240f24a25f4ee2b62df78be0ac271323c6968f322aba72c0f11082bd116980bc"} Nov 02 23:19:23.973117 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:23.973089 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" Nov 02 23:19:24.056413 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.056376 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5662cc68-8f5b-48f8-8ed9-858e5595524d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5662cc68-8f5b-48f8-8ed9-858e5595524d-istio-data\") pod \"5662cc68-8f5b-48f8-8ed9-858e5595524d\" (UID: \"5662cc68-8f5b-48f8-8ed9-858e5595524d\") " Nov 02 23:19:24.056599 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.056418 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5662cc68-8f5b-48f8-8ed9-858e5595524d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5662cc68-8f5b-48f8-8ed9-858e5595524d-workload-certs\") pod \"5662cc68-8f5b-48f8-8ed9-858e5595524d\" (UID: \"5662cc68-8f5b-48f8-8ed9-858e5595524d\") " Nov 02 23:19:24.056599 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.056465 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5662cc68-8f5b-48f8-8ed9-858e5595524d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5662cc68-8f5b-48f8-8ed9-858e5595524d-istio-envoy\") pod \"5662cc68-8f5b-48f8-8ed9-858e5595524d\" (UID: \"5662cc68-8f5b-48f8-8ed9-858e5595524d\") " Nov 02 23:19:24.056599 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.056524 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5662cc68-8f5b-48f8-8ed9-858e5595524d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5662cc68-8f5b-48f8-8ed9-858e5595524d-istio-podinfo\") pod \"5662cc68-8f5b-48f8-8ed9-858e5595524d\" (UID: \"5662cc68-8f5b-48f8-8ed9-858e5595524d\") " Nov 02 23:19:24.056756 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.056647 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5662cc68-8f5b-48f8-8ed9-858e5595524d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5662cc68-8f5b-48f8-8ed9-858e5595524d-istiod-ca-cert\") pod \"5662cc68-8f5b-48f8-8ed9-858e5595524d\" (UID: \"5662cc68-8f5b-48f8-8ed9-858e5595524d\") " Nov 02 23:19:24.056756 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.056665 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5662cc68-8f5b-48f8-8ed9-858e5595524d-istio-data" pod "5662cc68-8f5b-48f8-8ed9-858e5595524d" (UID: "5662cc68-8f5b-48f8-8ed9-858e5595524d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:24.056756 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.056715 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5662cc68-8f5b-48f8-8ed9-858e5595524d-kube-api-access-zx8b5\" (UniqueName: \"kubernetes.io/projected/5662cc68-8f5b-48f8-8ed9-858e5595524d-kube-api-access-zx8b5\") pod \"5662cc68-8f5b-48f8-8ed9-858e5595524d\" (UID: \"5662cc68-8f5b-48f8-8ed9-858e5595524d\") " Nov 02 23:19:24.056935 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.056791 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5662cc68-8f5b-48f8-8ed9-858e5595524d-workload-certs" pod "5662cc68-8f5b-48f8-8ed9-858e5595524d" (UID: "5662cc68-8f5b-48f8-8ed9-858e5595524d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:24.056935 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.056820 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5662cc68-8f5b-48f8-8ed9-858e5595524d-istio-token\" (UniqueName: \"kubernetes.io/projected/5662cc68-8f5b-48f8-8ed9-858e5595524d-istio-token\") pod \"5662cc68-8f5b-48f8-8ed9-858e5595524d\" (UID: \"5662cc68-8f5b-48f8-8ed9-858e5595524d\") " Nov 02 23:19:24.056935 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.056913 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5662cc68-8f5b-48f8-8ed9-858e5595524d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5662cc68-8f5b-48f8-8ed9-858e5595524d-workload-socket\") pod \"5662cc68-8f5b-48f8-8ed9-858e5595524d\" (UID: \"5662cc68-8f5b-48f8-8ed9-858e5595524d\") " Nov 02 23:19:24.057105 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.057042 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5662cc68-8f5b-48f8-8ed9-858e5595524d-istiod-ca-cert" pod "5662cc68-8f5b-48f8-8ed9-858e5595524d" (UID: "5662cc68-8f5b-48f8-8ed9-858e5595524d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:19:24.057180 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.057160 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5662cc68-8f5b-48f8-8ed9-858e5595524d-workload-socket" pod "5662cc68-8f5b-48f8-8ed9-858e5595524d" (UID: "5662cc68-8f5b-48f8-8ed9-858e5595524d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:24.057339 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.057290 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5662cc68-8f5b-48f8-8ed9-858e5595524d-credential-socket" pod "5662cc68-8f5b-48f8-8ed9-858e5595524d" (UID: "5662cc68-8f5b-48f8-8ed9-858e5595524d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:24.057499 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.057475 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5662cc68-8f5b-48f8-8ed9-858e5595524d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5662cc68-8f5b-48f8-8ed9-858e5595524d-credential-socket\") pod \"5662cc68-8f5b-48f8-8ed9-858e5595524d\" (UID: \"5662cc68-8f5b-48f8-8ed9-858e5595524d\") " Nov 02 23:19:24.058157 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.058136 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5662cc68-8f5b-48f8-8ed9-858e5595524d-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:24.058198 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.058164 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5662cc68-8f5b-48f8-8ed9-858e5595524d-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:24.058198 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.058180 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5662cc68-8f5b-48f8-8ed9-858e5595524d-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:24.058267 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.058195 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5662cc68-8f5b-48f8-8ed9-858e5595524d-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:24.058267 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.058209 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5662cc68-8f5b-48f8-8ed9-858e5595524d-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:24.058816 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.058794 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5662cc68-8f5b-48f8-8ed9-858e5595524d-istio-podinfo" pod "5662cc68-8f5b-48f8-8ed9-858e5595524d" (UID: "5662cc68-8f5b-48f8-8ed9-858e5595524d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:19:24.059002 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.058979 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5662cc68-8f5b-48f8-8ed9-858e5595524d-kube-api-access-zx8b5" pod "5662cc68-8f5b-48f8-8ed9-858e5595524d" (UID: "5662cc68-8f5b-48f8-8ed9-858e5595524d"). InnerVolumeSpecName "kube-api-access-zx8b5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:19:24.059062 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.059044 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5662cc68-8f5b-48f8-8ed9-858e5595524d-istio-envoy" pod "5662cc68-8f5b-48f8-8ed9-858e5595524d" (UID: "5662cc68-8f5b-48f8-8ed9-858e5595524d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:24.059209 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.059184 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5662cc68-8f5b-48f8-8ed9-858e5595524d-istio-token" pod "5662cc68-8f5b-48f8-8ed9-858e5595524d" (UID: "5662cc68-8f5b-48f8-8ed9-858e5595524d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:19:24.159000 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.158858 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx8b5\" (UniqueName: \"kubernetes.io/projected/5662cc68-8f5b-48f8-8ed9-858e5595524d-kube-api-access-zx8b5\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:24.159000 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.158917 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5662cc68-8f5b-48f8-8ed9-858e5595524d-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:24.159000 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.158935 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5662cc68-8f5b-48f8-8ed9-858e5595524d-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:24.159000 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.158948 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5662cc68-8f5b-48f8-8ed9-858e5595524d-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:24.419856 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.419778 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-818862-9f98cbf95-vwbzx"] Nov 02 23:19:24.420383 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.420369 5076 cpu_manager.go:414] "containerMap: removing container" podUID="ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b" containerName="istio-proxy" Nov 02 23:19:24.420427 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.420384 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b" containerName="istio-proxy" Nov 02 23:19:24.420427 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.420395 5076 cpu_manager.go:414] "containerMap: removing container" podUID="ca62790c-9f83-45df-9bb8-6360b271dab0" containerName="istio-proxy" Nov 02 23:19:24.420427 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.420400 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca62790c-9f83-45df-9bb8-6360b271dab0" containerName="istio-proxy" Nov 02 23:19:24.420427 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.420409 5076 cpu_manager.go:414] "containerMap: removing container" podUID="0450c8f1-c6f6-4d65-a0a0-7e954133793d" containerName="istio-proxy" Nov 02 23:19:24.420427 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.420414 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0450c8f1-c6f6-4d65-a0a0-7e954133793d" containerName="istio-proxy" Nov 02 23:19:24.420427 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.420422 5076 cpu_manager.go:414] "containerMap: removing container" podUID="8dd885dc-9e61-4303-a916-f422ab5eb322" containerName="istio-proxy" Nov 02 23:19:24.420427 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.420426 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dd885dc-9e61-4303-a916-f422ab5eb322" containerName="istio-proxy" Nov 02 23:19:24.420629 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.420443 5076 cpu_manager.go:414] "containerMap: removing container" podUID="5662cc68-8f5b-48f8-8ed9-858e5595524d" containerName="istio-proxy" Nov 02 23:19:24.420629 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.420449 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5662cc68-8f5b-48f8-8ed9-858e5595524d" containerName="istio-proxy" Nov 02 23:19:24.420629 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.420528 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b" containerName="istio-proxy" Nov 02 23:19:24.420629 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.420537 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca62790c-9f83-45df-9bb8-6360b271dab0" containerName="istio-proxy" Nov 02 23:19:24.420629 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.420542 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="0450c8f1-c6f6-4d65-a0a0-7e954133793d" containerName="istio-proxy" Nov 02 23:19:24.420629 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.420550 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dd885dc-9e61-4303-a916-f422ab5eb322" containerName="istio-proxy" Nov 02 23:19:24.420629 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.420559 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="5662cc68-8f5b-48f8-8ed9-858e5595524d" containerName="istio-proxy" Nov 02 23:19:24.424356 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.424328 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-818862-9f98cbf95-vwbzx" Nov 02 23:19:24.475927 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.474007 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-818862\"" Nov 02 23:19:24.475927 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.474168 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-818862-dockercfg-db7j4\"" Nov 02 23:19:24.476127 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.475932 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-818862-9f98cbf95-vwbzx"] Nov 02 23:19:24.591915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.565797 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49aee3c8-3339-4424-a3da-b81c4731e66a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-818862-9f98cbf95-vwbzx\" (UID: \"49aee3c8-3339-4424-a3da-b81c4731e66a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-818862-9f98cbf95-vwbzx" Nov 02 23:19:24.591915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.565864 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gd7s\" (UniqueName: \"kubernetes.io/projected/49aee3c8-3339-4424-a3da-b81c4731e66a-kube-api-access-9gd7s\") pod \"payload-pre-processing-e2e-api-b-818862-9f98cbf95-vwbzx\" (UID: \"49aee3c8-3339-4424-a3da-b81c4731e66a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-818862-9f98cbf95-vwbzx" Nov 02 23:19:24.660692 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.660655 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-818862-6f5ffd459f-q9ddl"] Nov 02 23:19:24.667787 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.667687 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49aee3c8-3339-4424-a3da-b81c4731e66a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-818862-9f98cbf95-vwbzx\" (UID: \"49aee3c8-3339-4424-a3da-b81c4731e66a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-818862-9f98cbf95-vwbzx" Nov 02 23:19:24.667787 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.667756 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gd7s\" (UniqueName: \"kubernetes.io/projected/49aee3c8-3339-4424-a3da-b81c4731e66a-kube-api-access-9gd7s\") pod \"payload-pre-processing-e2e-api-b-818862-9f98cbf95-vwbzx\" (UID: \"49aee3c8-3339-4424-a3da-b81c4731e66a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-818862-9f98cbf95-vwbzx" Nov 02 23:19:24.669266 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.669221 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49aee3c8-3339-4424-a3da-b81c4731e66a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-818862-9f98cbf95-vwbzx\" (UID: \"49aee3c8-3339-4424-a3da-b81c4731e66a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-818862-9f98cbf95-vwbzx" Nov 02 23:19:24.669771 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.669750 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-818862-6f5ffd459f-q9ddl" Nov 02 23:19:24.673518 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.673447 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-818862-6f5ffd459f-q9ddl"] Nov 02 23:19:24.723927 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.723878 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gd7s\" (UniqueName: \"kubernetes.io/projected/49aee3c8-3339-4424-a3da-b81c4731e66a-kube-api-access-9gd7s\") pod \"payload-pre-processing-e2e-api-b-818862-9f98cbf95-vwbzx\" (UID: \"49aee3c8-3339-4424-a3da-b81c4731e66a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-818862-9f98cbf95-vwbzx" Nov 02 23:19:24.743213 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.743179 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-818862-9f98cbf95-vwbzx" Nov 02 23:19:24.768843 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.768808 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d833e3b-de7a-42b4-8d36-0d8f1b7659ca-plugins-config-volume\") pod \"payload-processing-e2e-api-b-818862-6f5ffd459f-q9ddl\" (UID: \"6d833e3b-de7a-42b4-8d36-0d8f1b7659ca\") " pod="openshift-ingress/payload-processing-e2e-api-b-818862-6f5ffd459f-q9ddl" Nov 02 23:19:24.768976 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.768869 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm7dd\" (UniqueName: \"kubernetes.io/projected/6d833e3b-de7a-42b4-8d36-0d8f1b7659ca-kube-api-access-dm7dd\") pod \"payload-processing-e2e-api-b-818862-6f5ffd459f-q9ddl\" (UID: \"6d833e3b-de7a-42b4-8d36-0d8f1b7659ca\") " pod="openshift-ingress/payload-processing-e2e-api-b-818862-6f5ffd459f-q9ddl" Nov 02 23:19:24.873637 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.872316 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d833e3b-de7a-42b4-8d36-0d8f1b7659ca-plugins-config-volume\") pod \"payload-processing-e2e-api-b-818862-6f5ffd459f-q9ddl\" (UID: \"6d833e3b-de7a-42b4-8d36-0d8f1b7659ca\") " pod="openshift-ingress/payload-processing-e2e-api-b-818862-6f5ffd459f-q9ddl" Nov 02 23:19:24.873637 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.872413 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm7dd\" (UniqueName: \"kubernetes.io/projected/6d833e3b-de7a-42b4-8d36-0d8f1b7659ca-kube-api-access-dm7dd\") pod \"payload-processing-e2e-api-b-818862-6f5ffd459f-q9ddl\" (UID: \"6d833e3b-de7a-42b4-8d36-0d8f1b7659ca\") " pod="openshift-ingress/payload-processing-e2e-api-b-818862-6f5ffd459f-q9ddl" Nov 02 23:19:24.873879 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.873781 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d833e3b-de7a-42b4-8d36-0d8f1b7659ca-plugins-config-volume\") pod \"payload-processing-e2e-api-b-818862-6f5ffd459f-q9ddl\" (UID: \"6d833e3b-de7a-42b4-8d36-0d8f1b7659ca\") " pod="openshift-ingress/payload-processing-e2e-api-b-818862-6f5ffd459f-q9ddl" Nov 02 23:19:24.902079 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.902044 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" Nov 02 23:19:24.902489 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.902030 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw" event={"ID":"5662cc68-8f5b-48f8-8ed9-858e5595524d","Type":"ContainerDied","Data":"bbddb1980d2d15a3a360886c5b4f529c0b1cfa77683747955f4e782f24f9d585"} Nov 02 23:19:24.902489 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.902237 5076 scope.go:122] "RemoveContainer" containerID="240f24a25f4ee2b62df78be0ac271323c6968f322aba72c0f11082bd116980bc" Nov 02 23:19:24.926801 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.926745 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm7dd\" (UniqueName: \"kubernetes.io/projected/6d833e3b-de7a-42b4-8d36-0d8f1b7659ca-kube-api-access-dm7dd\") pod \"payload-processing-e2e-api-b-818862-6f5ffd459f-q9ddl\" (UID: \"6d833e3b-de7a-42b4-8d36-0d8f1b7659ca\") " pod="openshift-ingress/payload-processing-e2e-api-b-818862-6f5ffd459f-q9ddl" Nov 02 23:19:24.993145 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:24.993108 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-818862-6f5ffd459f-q9ddl" Nov 02 23:19:25.062213 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:25.061507 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw"] Nov 02 23:19:25.067373 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:25.067343 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-pxhgw"] Nov 02 23:19:25.149861 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:25.149832 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0450c8f1-c6f6-4d65-a0a0-7e954133793d" path="/var/lib/kubelet/pods/0450c8f1-c6f6-4d65-a0a0-7e954133793d/volumes" Nov 02 23:19:25.150429 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:25.150412 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5662cc68-8f5b-48f8-8ed9-858e5595524d" path="/var/lib/kubelet/pods/5662cc68-8f5b-48f8-8ed9-858e5595524d/volumes" Nov 02 23:19:25.150870 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:25.150856 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dd885dc-9e61-4303-a916-f422ab5eb322" path="/var/lib/kubelet/pods/8dd885dc-9e61-4303-a916-f422ab5eb322/volumes" Nov 02 23:19:25.151587 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:25.151570 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca62790c-9f83-45df-9bb8-6360b271dab0" path="/var/lib/kubelet/pods/ca62790c-9f83-45df-9bb8-6360b271dab0/volumes" Nov 02 23:19:25.152055 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:25.152034 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b" path="/var/lib/kubelet/pods/ccc8ac87-3bdb-4fad-8bb5-5ccefd0a357b/volumes" Nov 02 23:19:25.163626 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:25.163604 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-818862-9f98cbf95-vwbzx"] Nov 02 23:19:25.165167 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:19:25.165138 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49aee3c8_3339_4424_a3da_b81c4731e66a.slice/crio-d4a6c114b57cc298e9c7b2322f5fd546a59a90a974d2a67adc9c0461283131cf WatchSource:0}: Error finding container d4a6c114b57cc298e9c7b2322f5fd546a59a90a974d2a67adc9c0461283131cf: Status 404 returned error can't find the container with id d4a6c114b57cc298e9c7b2322f5fd546a59a90a974d2a67adc9c0461283131cf Nov 02 23:19:25.362985 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:25.362955 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-818862-6f5ffd459f-q9ddl"] Nov 02 23:19:25.363676 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:19:25.363646 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d833e3b_de7a_42b4_8d36_0d8f1b7659ca.slice/crio-f5058a849c3ba8dd57142926c33e7e4a4d0ee38707028c64f9b03a5a9608a422 WatchSource:0}: Error finding container f5058a849c3ba8dd57142926c33e7e4a4d0ee38707028c64f9b03a5a9608a422: Status 404 returned error can't find the container with id f5058a849c3ba8dd57142926c33e7e4a4d0ee38707028c64f9b03a5a9608a422 Nov 02 23:19:25.911103 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:25.911053 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-818862-9f98cbf95-vwbzx" event={"ID":"49aee3c8-3339-4424-a3da-b81c4731e66a","Type":"ContainerStarted","Data":"83371c0399feb68d40f17c5e66d967ad6ab961bdbc2f2608249bcb442db72766"} Nov 02 23:19:25.911590 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:25.911275 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-818862-9f98cbf95-vwbzx" Nov 02 23:19:25.911590 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:25.911306 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-818862-9f98cbf95-vwbzx" event={"ID":"49aee3c8-3339-4424-a3da-b81c4731e66a","Type":"ContainerStarted","Data":"d4a6c114b57cc298e9c7b2322f5fd546a59a90a974d2a67adc9c0461283131cf"} Nov 02 23:19:25.913868 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:25.913842 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-818862-6f5ffd459f-q9ddl" event={"ID":"6d833e3b-de7a-42b4-8d36-0d8f1b7659ca","Type":"ContainerStarted","Data":"6e5b48632097918c4c2a0c77e51a1cdf9f66a3c1a96abefde0c285eab43f575d"} Nov 02 23:19:25.914035 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:25.913886 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-818862-6f5ffd459f-q9ddl" event={"ID":"6d833e3b-de7a-42b4-8d36-0d8f1b7659ca","Type":"ContainerStarted","Data":"f5058a849c3ba8dd57142926c33e7e4a4d0ee38707028c64f9b03a5a9608a422"} Nov 02 23:19:25.914035 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:25.913945 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-818862-6f5ffd459f-q9ddl" Nov 02 23:19:26.022533 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:26.022475 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-818862-9f98cbf95-vwbzx" podStartSLOduration=2.022460969 podStartE2EDuration="2.022460969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:19:26.019210401 +0000 UTC m=+1614.987742188" watchObservedRunningTime="2026-11-02 23:19:26.022460969 +0000 UTC m=+1614.990992752" Nov 02 23:19:26.127565 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:26.127471 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-818862-6f5ffd459f-q9ddl" podStartSLOduration=2.127449138 podStartE2EDuration="2.127449138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:19:26.126073832 +0000 UTC m=+1615.094605641" watchObservedRunningTime="2026-11-02 23:19:26.127449138 +0000 UTC m=+1615.095980925" Nov 02 23:19:34.802496 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:34.802449 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729"] Nov 02 23:19:34.808939 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:34.808917 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:19:35.093480 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:35.093398 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-a69f65-openshift-default-dockercfg-qdxrd\"" Nov 02 23:19:35.101964 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:35.100953 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729"] Nov 02 23:19:35.101964 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:35.101631 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd184b6f-3901-413f-ad6a-c7271183796f-istio-data\") pod \"e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729\" (UID: \"dd184b6f-3901-413f-ad6a-c7271183796f\") " pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:19:35.101964 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:35.101690 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd184b6f-3901-413f-ad6a-c7271183796f-istio-podinfo\") pod \"e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729\" (UID: \"dd184b6f-3901-413f-ad6a-c7271183796f\") " pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:19:35.101964 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:35.101721 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd184b6f-3901-413f-ad6a-c7271183796f-istiod-ca-cert\") pod \"e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729\" (UID: \"dd184b6f-3901-413f-ad6a-c7271183796f\") " pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:19:35.101964 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:35.101784 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp72b\" (UniqueName: \"kubernetes.io/projected/dd184b6f-3901-413f-ad6a-c7271183796f-kube-api-access-lp72b\") pod \"e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729\" (UID: \"dd184b6f-3901-413f-ad6a-c7271183796f\") " pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:19:35.101964 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:35.101806 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd184b6f-3901-413f-ad6a-c7271183796f-istio-token\") pod \"e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729\" (UID: \"dd184b6f-3901-413f-ad6a-c7271183796f\") " pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:19:35.102943 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:35.102491 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd184b6f-3901-413f-ad6a-c7271183796f-istio-envoy\") pod \"e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729\" (UID: \"dd184b6f-3901-413f-ad6a-c7271183796f\") " pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:19:35.102943 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:35.102571 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd184b6f-3901-413f-ad6a-c7271183796f-credential-socket\") pod \"e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729\" (UID: \"dd184b6f-3901-413f-ad6a-c7271183796f\") " pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:19:35.102943 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:35.102617 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd184b6f-3901-413f-ad6a-c7271183796f-workload-socket\") pod \"e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729\" (UID: \"dd184b6f-3901-413f-ad6a-c7271183796f\") " pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:19:35.102943 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:35.102671 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd184b6f-3901-413f-ad6a-c7271183796f-workload-certs\") pod \"e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729\" (UID: \"dd184b6f-3901-413f-ad6a-c7271183796f\") " pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:19:35.203542 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:35.203495 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd184b6f-3901-413f-ad6a-c7271183796f-istio-envoy\") pod \"e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729\" (UID: \"dd184b6f-3901-413f-ad6a-c7271183796f\") " pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:19:35.203695 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:35.203550 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd184b6f-3901-413f-ad6a-c7271183796f-credential-socket\") pod \"e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729\" (UID: \"dd184b6f-3901-413f-ad6a-c7271183796f\") " pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:19:35.203695 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:35.203576 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd184b6f-3901-413f-ad6a-c7271183796f-workload-socket\") pod \"e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729\" (UID: \"dd184b6f-3901-413f-ad6a-c7271183796f\") " pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:19:35.203808 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:35.203718 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd184b6f-3901-413f-ad6a-c7271183796f-workload-certs\") pod \"e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729\" (UID: \"dd184b6f-3901-413f-ad6a-c7271183796f\") " pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:19:35.203872 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:35.203803 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd184b6f-3901-413f-ad6a-c7271183796f-istio-data\") pod \"e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729\" (UID: \"dd184b6f-3901-413f-ad6a-c7271183796f\") " pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:19:35.203872 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:35.203859 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd184b6f-3901-413f-ad6a-c7271183796f-istio-podinfo\") pod \"e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729\" (UID: \"dd184b6f-3901-413f-ad6a-c7271183796f\") " pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:19:35.203998 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:35.203902 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd184b6f-3901-413f-ad6a-c7271183796f-istiod-ca-cert\") pod \"e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729\" (UID: \"dd184b6f-3901-413f-ad6a-c7271183796f\") " pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:19:35.203998 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:35.203971 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp72b\" (UniqueName: \"kubernetes.io/projected/dd184b6f-3901-413f-ad6a-c7271183796f-kube-api-access-lp72b\") pod \"e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729\" (UID: \"dd184b6f-3901-413f-ad6a-c7271183796f\") " pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:19:35.204110 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:35.203995 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd184b6f-3901-413f-ad6a-c7271183796f-istio-token\") pod \"e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729\" (UID: \"dd184b6f-3901-413f-ad6a-c7271183796f\") " pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:19:35.204110 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:35.204007 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd184b6f-3901-413f-ad6a-c7271183796f-credential-socket\") pod \"e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729\" (UID: \"dd184b6f-3901-413f-ad6a-c7271183796f\") " pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:19:35.204278 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:35.204252 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd184b6f-3901-413f-ad6a-c7271183796f-istio-data\") pod \"e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729\" (UID: \"dd184b6f-3901-413f-ad6a-c7271183796f\") " pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:19:35.204278 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:35.204265 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd184b6f-3901-413f-ad6a-c7271183796f-workload-certs\") pod \"e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729\" (UID: \"dd184b6f-3901-413f-ad6a-c7271183796f\") " pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:19:35.204688 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:35.204637 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd184b6f-3901-413f-ad6a-c7271183796f-istiod-ca-cert\") pod \"e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729\" (UID: \"dd184b6f-3901-413f-ad6a-c7271183796f\") " pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:19:35.204688 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:35.204662 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd184b6f-3901-413f-ad6a-c7271183796f-workload-socket\") pod \"e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729\" (UID: \"dd184b6f-3901-413f-ad6a-c7271183796f\") " pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:19:35.206061 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:35.206036 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd184b6f-3901-413f-ad6a-c7271183796f-istio-envoy\") pod \"e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729\" (UID: \"dd184b6f-3901-413f-ad6a-c7271183796f\") " pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:19:35.206574 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:35.206557 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd184b6f-3901-413f-ad6a-c7271183796f-istio-podinfo\") pod \"e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729\" (UID: \"dd184b6f-3901-413f-ad6a-c7271183796f\") " pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:19:35.285800 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:35.285730 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp72b\" (UniqueName: \"kubernetes.io/projected/dd184b6f-3901-413f-ad6a-c7271183796f-kube-api-access-lp72b\") pod \"e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729\" (UID: \"dd184b6f-3901-413f-ad6a-c7271183796f\") " pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:19:35.287636 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:35.287605 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd184b6f-3901-413f-ad6a-c7271183796f-istio-token\") pod \"e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729\" (UID: \"dd184b6f-3901-413f-ad6a-c7271183796f\") " pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:19:35.422313 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:35.422218 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:19:35.843470 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:35.843433 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729"] Nov 02 23:19:35.879191 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:19:35.879165 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd184b6f_3901_413f_ad6a_c7271183796f.slice/crio-d0d3d35661461734718db94e2aeaf94befa9f0a3f94707c416f45ffe8b05c604 WatchSource:0}: Error finding container d0d3d35661461734718db94e2aeaf94befa9f0a3f94707c416f45ffe8b05c604: Status 404 returned error can't find the container with id d0d3d35661461734718db94e2aeaf94befa9f0a3f94707c416f45ffe8b05c604 Nov 02 23:19:35.881842 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:35.881812 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:19:35.881938 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:35.881876 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:19:35.881938 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:35.881918 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:19:36.127202 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:36.127105 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" event={"ID":"dd184b6f-3901-413f-ad6a-c7271183796f","Type":"ContainerStarted","Data":"cd88cb2e172b9fbd8f8043805751be2d839b447e7726bcc933dbecb51fbb208e"} Nov 02 23:19:36.127202 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:36.127164 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" event={"ID":"dd184b6f-3901-413f-ad6a-c7271183796f","Type":"ContainerStarted","Data":"d0d3d35661461734718db94e2aeaf94befa9f0a3f94707c416f45ffe8b05c604"} Nov 02 23:19:36.242600 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:36.242543 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" podStartSLOduration=2.242528096 podStartE2EDuration="2.242528096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:19:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:19:36.240859871 +0000 UTC m=+1625.209391655" watchObservedRunningTime="2026-11-02 23:19:36.242528096 +0000 UTC m=+1625.211059879" Nov 02 23:19:36.424808 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:36.424581 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:19:36.432689 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:36.432660 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:19:36.924236 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:36.924206 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-818862-6f5ffd459f-q9ddl" Nov 02 23:19:36.924631 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:36.924280 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-818862-9f98cbf95-vwbzx" Nov 02 23:19:37.134192 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:37.134160 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:19:37.135025 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:37.135005 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:19:39.886087 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:19:39.886053 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:19:43.705060 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.705027 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js"] Nov 02 23:19:43.710433 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.710411 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" Nov 02 23:19:43.752292 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.752260 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr"] Nov 02 23:19:43.756345 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.756328 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" Nov 02 23:19:43.796625 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.796587 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e953257b-openshift-default-dockercfg-9qzkk\"" Nov 02 23:19:43.802091 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.802058 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js"] Nov 02 23:19:43.853767 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.853724 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr"] Nov 02 23:19:43.864011 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.863979 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21678735-6a1a-4654-8b9d-b979a78520ff-istio-token\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js\" (UID: \"21678735-6a1a-4654-8b9d-b979a78520ff\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" Nov 02 23:19:43.864168 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.864021 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21678735-6a1a-4654-8b9d-b979a78520ff-credential-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js\" (UID: \"21678735-6a1a-4654-8b9d-b979a78520ff\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" Nov 02 23:19:43.864168 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.864053 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21678735-6a1a-4654-8b9d-b979a78520ff-istio-data\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js\" (UID: \"21678735-6a1a-4654-8b9d-b979a78520ff\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" Nov 02 23:19:43.864271 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.864174 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-istio-data\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr\" (UID: \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" Nov 02 23:19:43.864271 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.864229 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-istio-token\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr\" (UID: \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" Nov 02 23:19:43.864378 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.864316 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-istio-envoy\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr\" (UID: \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" Nov 02 23:19:43.864378 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.864356 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-credential-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr\" (UID: \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" Nov 02 23:19:43.864479 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.864394 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21678735-6a1a-4654-8b9d-b979a78520ff-istio-envoy\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js\" (UID: \"21678735-6a1a-4654-8b9d-b979a78520ff\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" Nov 02 23:19:43.864479 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.864424 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-istio-podinfo\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr\" (UID: \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" Nov 02 23:19:43.864479 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.864471 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-istiod-ca-cert\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr\" (UID: \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" Nov 02 23:19:43.864624 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.864579 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt522\" (UniqueName: \"kubernetes.io/projected/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-kube-api-access-bt522\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr\" (UID: \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" Nov 02 23:19:43.864624 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.864614 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21678735-6a1a-4654-8b9d-b979a78520ff-workload-certs\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js\" (UID: \"21678735-6a1a-4654-8b9d-b979a78520ff\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" Nov 02 23:19:43.864720 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.864637 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-workload-certs\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr\" (UID: \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" Nov 02 23:19:43.864720 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.864680 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp5zc\" (UniqueName: \"kubernetes.io/projected/21678735-6a1a-4654-8b9d-b979a78520ff-kube-api-access-hp5zc\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js\" (UID: \"21678735-6a1a-4654-8b9d-b979a78520ff\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" Nov 02 23:19:43.864720 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.864708 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-workload-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr\" (UID: \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" Nov 02 23:19:43.864869 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.864734 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21678735-6a1a-4654-8b9d-b979a78520ff-istio-podinfo\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js\" (UID: \"21678735-6a1a-4654-8b9d-b979a78520ff\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" Nov 02 23:19:43.864869 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.864785 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21678735-6a1a-4654-8b9d-b979a78520ff-istiod-ca-cert\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js\" (UID: \"21678735-6a1a-4654-8b9d-b979a78520ff\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" Nov 02 23:19:43.864869 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.864827 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21678735-6a1a-4654-8b9d-b979a78520ff-workload-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js\" (UID: \"21678735-6a1a-4654-8b9d-b979a78520ff\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" Nov 02 23:19:43.966552 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.966461 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-istio-podinfo\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr\" (UID: \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" Nov 02 23:19:43.966552 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.966516 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-istiod-ca-cert\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr\" (UID: \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" Nov 02 23:19:43.966777 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.966592 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt522\" (UniqueName: \"kubernetes.io/projected/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-kube-api-access-bt522\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr\" (UID: \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" Nov 02 23:19:43.966777 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.966613 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21678735-6a1a-4654-8b9d-b979a78520ff-workload-certs\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js\" (UID: \"21678735-6a1a-4654-8b9d-b979a78520ff\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" Nov 02 23:19:43.966777 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.966628 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-workload-certs\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr\" (UID: \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" Nov 02 23:19:43.966777 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.966653 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp5zc\" (UniqueName: \"kubernetes.io/projected/21678735-6a1a-4654-8b9d-b979a78520ff-kube-api-access-hp5zc\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js\" (UID: \"21678735-6a1a-4654-8b9d-b979a78520ff\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" Nov 02 23:19:43.966777 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.966670 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-workload-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr\" (UID: \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" Nov 02 23:19:43.966777 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.966687 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21678735-6a1a-4654-8b9d-b979a78520ff-istio-podinfo\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js\" (UID: \"21678735-6a1a-4654-8b9d-b979a78520ff\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" Nov 02 23:19:43.966777 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.966728 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21678735-6a1a-4654-8b9d-b979a78520ff-istiod-ca-cert\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js\" (UID: \"21678735-6a1a-4654-8b9d-b979a78520ff\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" Nov 02 23:19:43.966777 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.966765 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21678735-6a1a-4654-8b9d-b979a78520ff-workload-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js\" (UID: \"21678735-6a1a-4654-8b9d-b979a78520ff\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" Nov 02 23:19:43.967289 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.966840 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21678735-6a1a-4654-8b9d-b979a78520ff-istio-token\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js\" (UID: \"21678735-6a1a-4654-8b9d-b979a78520ff\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" Nov 02 23:19:43.967289 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.966869 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21678735-6a1a-4654-8b9d-b979a78520ff-credential-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js\" (UID: \"21678735-6a1a-4654-8b9d-b979a78520ff\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" Nov 02 23:19:43.967289 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.966914 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21678735-6a1a-4654-8b9d-b979a78520ff-istio-data\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js\" (UID: \"21678735-6a1a-4654-8b9d-b979a78520ff\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" Nov 02 23:19:43.967289 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.966947 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-istio-data\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr\" (UID: \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" Nov 02 23:19:43.967289 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.966989 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-istio-token\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr\" (UID: \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" Nov 02 23:19:43.967289 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.967030 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-workload-certs\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr\" (UID: \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" Nov 02 23:19:43.967289 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.967033 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-istio-envoy\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr\" (UID: \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" Nov 02 23:19:43.967289 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.967096 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-workload-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr\" (UID: \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" Nov 02 23:19:43.967289 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.967129 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-credential-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr\" (UID: \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" Nov 02 23:19:43.967289 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.967180 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21678735-6a1a-4654-8b9d-b979a78520ff-istio-envoy\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js\" (UID: \"21678735-6a1a-4654-8b9d-b979a78520ff\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" Nov 02 23:19:43.967793 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.967366 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21678735-6a1a-4654-8b9d-b979a78520ff-workload-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js\" (UID: \"21678735-6a1a-4654-8b9d-b979a78520ff\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" Nov 02 23:19:43.967793 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.967372 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-istiod-ca-cert\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr\" (UID: \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" Nov 02 23:19:43.967793 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.967517 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-istio-data\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr\" (UID: \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" Nov 02 23:19:43.967793 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.967627 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21678735-6a1a-4654-8b9d-b979a78520ff-credential-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js\" (UID: \"21678735-6a1a-4654-8b9d-b979a78520ff\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" Nov 02 23:19:43.967793 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.967081 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21678735-6a1a-4654-8b9d-b979a78520ff-workload-certs\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js\" (UID: \"21678735-6a1a-4654-8b9d-b979a78520ff\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" Nov 02 23:19:43.967793 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.967783 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21678735-6a1a-4654-8b9d-b979a78520ff-istio-data\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js\" (UID: \"21678735-6a1a-4654-8b9d-b979a78520ff\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" Nov 02 23:19:43.968095 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.967880 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-credential-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr\" (UID: \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" Nov 02 23:19:43.968095 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.968003 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21678735-6a1a-4654-8b9d-b979a78520ff-istiod-ca-cert\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js\" (UID: \"21678735-6a1a-4654-8b9d-b979a78520ff\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" Nov 02 23:19:43.969364 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.969339 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-istio-envoy\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr\" (UID: \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" Nov 02 23:19:43.969583 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.969548 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21678735-6a1a-4654-8b9d-b979a78520ff-istio-podinfo\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js\" (UID: \"21678735-6a1a-4654-8b9d-b979a78520ff\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" Nov 02 23:19:43.969767 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.969745 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21678735-6a1a-4654-8b9d-b979a78520ff-istio-envoy\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js\" (UID: \"21678735-6a1a-4654-8b9d-b979a78520ff\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" Nov 02 23:19:43.969973 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:43.969957 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-istio-podinfo\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr\" (UID: \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" Nov 02 23:19:44.020909 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:44.020859 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21678735-6a1a-4654-8b9d-b979a78520ff-istio-token\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js\" (UID: \"21678735-6a1a-4654-8b9d-b979a78520ff\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" Nov 02 23:19:44.021171 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:44.021142 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp5zc\" (UniqueName: \"kubernetes.io/projected/21678735-6a1a-4654-8b9d-b979a78520ff-kube-api-access-hp5zc\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js\" (UID: \"21678735-6a1a-4654-8b9d-b979a78520ff\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" Nov 02 23:19:44.021368 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:44.021348 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt522\" (UniqueName: \"kubernetes.io/projected/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-kube-api-access-bt522\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr\" (UID: \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" Nov 02 23:19:44.024652 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:44.024628 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" Nov 02 23:19:44.029653 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:44.029634 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-istio-token\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr\" (UID: \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" Nov 02 23:19:44.069958 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:44.069919 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" Nov 02 23:19:44.461924 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:44.460871 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:19:44.461924 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:44.460987 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:19:44.461924 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:44.461036 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:19:44.471138 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:44.471107 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js"] Nov 02 23:19:44.512862 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:44.512823 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr"] Nov 02 23:19:44.521261 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:19:44.521226 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c011db9_8ca8_46bd_a9d5_e33b8ce1dfdd.slice/crio-d39420625dd5d56b7d9616cca1a1bc80865950dd74ed4eea7fad9752ef4916ff WatchSource:0}: Error finding container d39420625dd5d56b7d9616cca1a1bc80865950dd74ed4eea7fad9752ef4916ff: Status 404 returned error can't find the container with id d39420625dd5d56b7d9616cca1a1bc80865950dd74ed4eea7fad9752ef4916ff Nov 02 23:19:44.524001 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:44.523968 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:19:44.524107 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:44.524050 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:19:44.524107 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:44.524077 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:19:45.187052 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:45.187014 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" event={"ID":"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd","Type":"ContainerStarted","Data":"bca75fd4bc4583c7bbd11161cd219d69b53399e71a8f0b8d628b68ccc56e46e0"} Nov 02 23:19:45.187427 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:45.187069 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" event={"ID":"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd","Type":"ContainerStarted","Data":"d39420625dd5d56b7d9616cca1a1bc80865950dd74ed4eea7fad9752ef4916ff"} Nov 02 23:19:45.189353 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:45.189325 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" event={"ID":"21678735-6a1a-4654-8b9d-b979a78520ff","Type":"ContainerStarted","Data":"9be3896d0fac297b0bb0bf2099051b9a07bc34bba7c3a101a473f1754dee03c9"} Nov 02 23:19:45.189481 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:45.189375 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" event={"ID":"21678735-6a1a-4654-8b9d-b979a78520ff","Type":"ContainerStarted","Data":"a4e86138fa90fb7a44887610c22e5b557ca11c62e831fb00e8abef37ec9630be"} Nov 02 23:19:45.305795 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:45.305245 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" podStartSLOduration=2.30522428 podStartE2EDuration="2.30522428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:19:45.300118398 +0000 UTC m=+1634.268650196" watchObservedRunningTime="2026-11-02 23:19:45.30522428 +0000 UTC m=+1634.273756067" Nov 02 23:19:45.428643 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:45.426411 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" podStartSLOduration=2.426391665 podStartE2EDuration="2.426391665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:19:45.422029099 +0000 UTC m=+1634.390560875" watchObservedRunningTime="2026-11-02 23:19:45.426391665 +0000 UTC m=+1634.394923460" Nov 02 23:19:46.034992 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:46.034958 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" Nov 02 23:19:46.039031 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:46.039002 5076 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 02 23:19:46.039244 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:46.039218 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" podUID="21678735-6a1a-4654-8b9d-b979a78520ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 02 23:19:46.071208 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:46.070956 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" Nov 02 23:19:46.083868 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:46.083833 5076 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 02 23:19:46.084188 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:46.084138 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" podUID="6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 02 23:19:47.048366 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:47.048331 5076 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 02 23:19:47.048947 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:47.048912 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" podUID="21678735-6a1a-4654-8b9d-b979a78520ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 02 23:19:47.071026 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:47.070994 5076 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 02 23:19:47.071187 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:47.071049 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" podUID="6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 02 23:19:48.032916 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:48.029366 5076 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 02 23:19:48.032916 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:48.029439 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" podUID="21678735-6a1a-4654-8b9d-b979a78520ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 02 23:19:48.071386 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:48.071348 5076 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 02 23:19:48.071761 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:48.071401 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" podUID="6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 02 23:19:49.025912 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:49.025339 5076 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 02 23:19:49.025912 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:49.025463 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" podUID="21678735-6a1a-4654-8b9d-b979a78520ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 02 23:19:49.070629 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:49.070593 5076 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 02 23:19:49.070804 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:49.070643 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" podUID="6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 02 23:19:50.033914 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:50.030298 5076 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 02 23:19:50.033914 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:50.030365 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" podUID="21678735-6a1a-4654-8b9d-b979a78520ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 02 23:19:50.071103 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:50.071066 5076 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 02 23:19:50.071277 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:50.071121 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" podUID="6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 02 23:19:51.115021 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:51.113987 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" Nov 02 23:19:51.115021 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:51.114450 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" Nov 02 23:19:51.115568 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:51.115408 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" Nov 02 23:19:51.122019 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:51.121995 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" Nov 02 23:19:51.122562 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:51.122528 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" Nov 02 23:19:51.123182 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:51.123165 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" Nov 02 23:19:53.210443 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:53.210403 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-027689-5864dbbf67-j588d"] Nov 02 23:19:53.221329 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:53.217646 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-027689-5864dbbf67-j588d"] Nov 02 23:19:53.221329 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:53.217768 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-027689-5864dbbf67-j588d" Nov 02 23:19:53.273532 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:53.273501 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-027689-serving-cert\"" Nov 02 23:19:53.300561 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:53.300522 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrs8w\" (UniqueName: \"kubernetes.io/projected/105ffb43-f715-44c3-815f-452b3086133f-kube-api-access-zrs8w\") pod \"maas-api-e2e-ipp-a-027689-5864dbbf67-j588d\" (UID: \"105ffb43-f715-44c3-815f-452b3086133f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-027689-5864dbbf67-j588d" Nov 02 23:19:53.300868 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:53.300843 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/105ffb43-f715-44c3-815f-452b3086133f-metrics-certs\") pod \"maas-api-e2e-ipp-a-027689-5864dbbf67-j588d\" (UID: \"105ffb43-f715-44c3-815f-452b3086133f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-027689-5864dbbf67-j588d" Nov 02 23:19:53.301013 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:53.300928 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/105ffb43-f715-44c3-815f-452b3086133f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-027689-5864dbbf67-j588d\" (UID: \"105ffb43-f715-44c3-815f-452b3086133f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-027689-5864dbbf67-j588d" Nov 02 23:19:53.301075 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:53.301030 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/105ffb43-f715-44c3-815f-452b3086133f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-027689-5864dbbf67-j588d\" (UID: \"105ffb43-f715-44c3-815f-452b3086133f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-027689-5864dbbf67-j588d" Nov 02 23:19:53.418511 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:53.402341 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/105ffb43-f715-44c3-815f-452b3086133f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-027689-5864dbbf67-j588d\" (UID: \"105ffb43-f715-44c3-815f-452b3086133f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-027689-5864dbbf67-j588d" Nov 02 23:19:53.418511 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:53.402443 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/105ffb43-f715-44c3-815f-452b3086133f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-027689-5864dbbf67-j588d\" (UID: \"105ffb43-f715-44c3-815f-452b3086133f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-027689-5864dbbf67-j588d" Nov 02 23:19:53.418511 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:53.402544 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrs8w\" (UniqueName: \"kubernetes.io/projected/105ffb43-f715-44c3-815f-452b3086133f-kube-api-access-zrs8w\") pod \"maas-api-e2e-ipp-a-027689-5864dbbf67-j588d\" (UID: \"105ffb43-f715-44c3-815f-452b3086133f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-027689-5864dbbf67-j588d" Nov 02 23:19:53.418511 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:53.402736 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/105ffb43-f715-44c3-815f-452b3086133f-metrics-certs\") pod \"maas-api-e2e-ipp-a-027689-5864dbbf67-j588d\" (UID: \"105ffb43-f715-44c3-815f-452b3086133f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-027689-5864dbbf67-j588d" Nov 02 23:19:53.418511 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:53.404175 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/105ffb43-f715-44c3-815f-452b3086133f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-027689-5864dbbf67-j588d\" (UID: \"105ffb43-f715-44c3-815f-452b3086133f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-027689-5864dbbf67-j588d" Nov 02 23:19:53.418511 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:19:53.404617 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-027689-serving-cert: secret "maas-api-e2e-ipp-a-027689-serving-cert" not found Nov 02 23:19:53.418511 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:19:53.404713 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/105ffb43-f715-44c3-815f-452b3086133f-maas-api-tls podName:105ffb43-f715-44c3-815f-452b3086133f nodeName:}" failed. No retries permitted until 2026-11-02 23:19:53.904693461 +0000 UTC m=+1642.873225223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/105ffb43-f715-44c3-815f-452b3086133f-maas-api-tls") pod "maas-api-e2e-ipp-a-027689-5864dbbf67-j588d" (UID: "105ffb43-f715-44c3-815f-452b3086133f") : secret "maas-api-e2e-ipp-a-027689-serving-cert" not found Nov 02 23:19:53.418511 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:53.409578 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/105ffb43-f715-44c3-815f-452b3086133f-metrics-certs\") pod \"maas-api-e2e-ipp-a-027689-5864dbbf67-j588d\" (UID: \"105ffb43-f715-44c3-815f-452b3086133f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-027689-5864dbbf67-j588d" Nov 02 23:19:53.462418 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:53.462333 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrs8w\" (UniqueName: \"kubernetes.io/projected/105ffb43-f715-44c3-815f-452b3086133f-kube-api-access-zrs8w\") pod \"maas-api-e2e-ipp-a-027689-5864dbbf67-j588d\" (UID: \"105ffb43-f715-44c3-815f-452b3086133f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-027689-5864dbbf67-j588d" Nov 02 23:19:53.912220 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:53.912136 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/105ffb43-f715-44c3-815f-452b3086133f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-027689-5864dbbf67-j588d\" (UID: \"105ffb43-f715-44c3-815f-452b3086133f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-027689-5864dbbf67-j588d" Nov 02 23:19:53.914590 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:53.914563 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/105ffb43-f715-44c3-815f-452b3086133f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-027689-5864dbbf67-j588d\" (UID: \"105ffb43-f715-44c3-815f-452b3086133f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-027689-5864dbbf67-j588d" Nov 02 23:19:54.197095 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:54.197050 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-027689-5864dbbf67-j588d" Nov 02 23:19:54.663282 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:54.663250 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-027689-5864dbbf67-j588d"] Nov 02 23:19:54.664353 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:19:54.664323 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod105ffb43_f715_44c3_815f_452b3086133f.slice/crio-f94569f4a875a82174f8777471b3d6313ec4e11de226604b0bd58392444f076e WatchSource:0}: Error finding container f94569f4a875a82174f8777471b3d6313ec4e11de226604b0bd58392444f076e: Status 404 returned error can't find the container with id f94569f4a875a82174f8777471b3d6313ec4e11de226604b0bd58392444f076e Nov 02 23:19:55.331691 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:55.331647 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-027689-5864dbbf67-j588d" event={"ID":"105ffb43-f715-44c3-815f-452b3086133f","Type":"ContainerStarted","Data":"f94569f4a875a82174f8777471b3d6313ec4e11de226604b0bd58392444f076e"} Nov 02 23:19:56.341490 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:56.341451 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-027689-5864dbbf67-j588d" event={"ID":"105ffb43-f715-44c3-815f-452b3086133f","Type":"ContainerStarted","Data":"fc22217d4c450fe836112a0fd2924d35a434af8a9653aa1941c2629715050ca1"} Nov 02 23:19:56.341934 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:56.341521 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-027689-5864dbbf67-j588d" Nov 02 23:19:56.453155 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:56.453103 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-027689-5864dbbf67-j588d" podStartSLOduration=2.5291205039999998 podStartE2EDuration="3.453087937s" totalImagesPullingTime="923.967433ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:19:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:19:56.449766783 +0000 UTC m=+1645.418298575" watchObservedRunningTime="2026-11-02 23:19:56.453087937 +0000 UTC m=+1645.421619724" Nov 02 23:19:58.844152 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:58.844108 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs"] Nov 02 23:19:58.849534 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:58.849513 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" Nov 02 23:19:58.936718 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:58.936678 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs"] Nov 02 23:19:58.982808 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:58.982773 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-credential-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs\" (UID: \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" Nov 02 23:19:58.982808 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:58.982816 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-workload-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs\" (UID: \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" Nov 02 23:19:58.983069 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:58.982885 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-istio-podinfo\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs\" (UID: \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" Nov 02 23:19:58.983069 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:58.982954 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-istiod-ca-cert\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs\" (UID: \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" Nov 02 23:19:58.983069 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:58.983030 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-istio-data\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs\" (UID: \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" Nov 02 23:19:58.983222 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:58.983177 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-istio-envoy\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs\" (UID: \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" Nov 02 23:19:58.983301 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:58.983286 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-istio-token\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs\" (UID: \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" Nov 02 23:19:58.983366 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:58.983348 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8gpb\" (UniqueName: \"kubernetes.io/projected/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-kube-api-access-f8gpb\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs\" (UID: \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" Nov 02 23:19:58.983407 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:58.983384 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-workload-certs\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs\" (UID: \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" Nov 02 23:19:59.085049 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:59.085007 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-istio-podinfo\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs\" (UID: \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" Nov 02 23:19:59.085049 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:59.085055 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-istiod-ca-cert\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs\" (UID: \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" Nov 02 23:19:59.085294 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:59.085171 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-istio-data\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs\" (UID: \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" Nov 02 23:19:59.085294 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:59.085278 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-istio-envoy\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs\" (UID: \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" Nov 02 23:19:59.085406 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:59.085339 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-istio-token\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs\" (UID: \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" Nov 02 23:19:59.085406 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:59.085384 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8gpb\" (UniqueName: \"kubernetes.io/projected/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-kube-api-access-f8gpb\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs\" (UID: \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" Nov 02 23:19:59.085516 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:59.085415 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-workload-certs\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs\" (UID: \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" Nov 02 23:19:59.085516 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:59.085494 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-credential-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs\" (UID: \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" Nov 02 23:19:59.085629 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:59.085529 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-workload-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs\" (UID: \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" Nov 02 23:19:59.085712 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:59.085664 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-istio-data\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs\" (UID: \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" Nov 02 23:19:59.085870 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:59.085845 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-workload-certs\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs\" (UID: \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" Nov 02 23:19:59.085978 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:59.085942 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-workload-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs\" (UID: \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" Nov 02 23:19:59.086035 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:59.085973 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-istiod-ca-cert\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs\" (UID: \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" Nov 02 23:19:59.086035 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:59.086006 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-credential-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs\" (UID: \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" Nov 02 23:19:59.087534 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:59.087503 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-istio-envoy\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs\" (UID: \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" Nov 02 23:19:59.087665 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:59.087648 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-istio-podinfo\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs\" (UID: \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" Nov 02 23:19:59.140851 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:59.140778 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-istio-token\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs\" (UID: \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" Nov 02 23:19:59.141008 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:59.140911 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8gpb\" (UniqueName: \"kubernetes.io/projected/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-kube-api-access-f8gpb\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs\" (UID: \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" Nov 02 23:19:59.163387 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:59.163356 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" Nov 02 23:19:59.590881 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:59.590847 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs"] Nov 02 23:19:59.591079 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:19:59.591056 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod984e5b3f_c720_44ac_9dfd_12ce2b4dc2bc.slice/crio-a3b6be039a284bddf057cba46ffc58fedb8e9446c432bce72cea40e2e6f6fa24 WatchSource:0}: Error finding container a3b6be039a284bddf057cba46ffc58fedb8e9446c432bce72cea40e2e6f6fa24: Status 404 returned error can't find the container with id a3b6be039a284bddf057cba46ffc58fedb8e9446c432bce72cea40e2e6f6fa24 Nov 02 23:19:59.594019 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:59.593985 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:19:59.594108 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:59.594051 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:19:59.594108 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:19:59.594079 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:20:00.369687 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:00.369645 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" event={"ID":"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc","Type":"ContainerStarted","Data":"6fad4738f09acc85561dcb3395f302dbc05dfba5ce94019d530b0acbb2c15da8"} Nov 02 23:20:00.370147 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:00.369716 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" event={"ID":"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc","Type":"ContainerStarted","Data":"a3b6be039a284bddf057cba46ffc58fedb8e9446c432bce72cea40e2e6f6fa24"} Nov 02 23:20:00.588652 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:00.588586 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" podStartSLOduration=2.588565008 podStartE2EDuration="2.588565008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:19:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:20:00.585542376 +0000 UTC m=+1649.554074162" watchObservedRunningTime="2026-11-02 23:20:00.588565008 +0000 UTC m=+1649.557096795" Nov 02 23:20:00.673707 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:00.673629 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7415-8368a1"] Nov 02 23:20:00.678199 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:00.678178 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7415-8368a1" Nov 02 23:20:00.734129 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:00.734093 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wbwlb\"" Nov 02 23:20:00.738335 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:00.738204 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7415-8368a1"] Nov 02 23:20:00.833917 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:00.809330 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p726w\" (UniqueName: \"kubernetes.io/projected/63ef8260-6155-4d11-a82c-bb48d232a4ac-kube-api-access-p726w\") pod \"test-curl-7415-8368a1\" (UID: \"63ef8260-6155-4d11-a82c-bb48d232a4ac\") " pod="opendatahub/test-curl-7415-8368a1" Nov 02 23:20:00.934131 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:00.934092 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p726w\" (UniqueName: \"kubernetes.io/projected/63ef8260-6155-4d11-a82c-bb48d232a4ac-kube-api-access-p726w\") pod \"test-curl-7415-8368a1\" (UID: \"63ef8260-6155-4d11-a82c-bb48d232a4ac\") " pod="opendatahub/test-curl-7415-8368a1" Nov 02 23:20:00.989029 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:00.988995 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p726w\" (UniqueName: \"kubernetes.io/projected/63ef8260-6155-4d11-a82c-bb48d232a4ac-kube-api-access-p726w\") pod \"test-curl-7415-8368a1\" (UID: \"63ef8260-6155-4d11-a82c-bb48d232a4ac\") " pod="opendatahub/test-curl-7415-8368a1" Nov 02 23:20:00.990587 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:00.990565 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7415-8368a1" Nov 02 23:20:01.164271 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:01.164208 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" Nov 02 23:20:01.166124 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:01.166090 5076 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 23:20:01.166261 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:01.166147 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" podUID="984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 23:20:01.398911 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:01.398875 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7415-8368a1"] Nov 02 23:20:01.400275 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:20:01.400246 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63ef8260_6155_4d11_a82c_bb48d232a4ac.slice/crio-5b60db6f5c8d7b1271ed5b2417cd8dc567ff51561c1a81a66f8608d90e6ea9d3 WatchSource:0}: Error finding container 5b60db6f5c8d7b1271ed5b2417cd8dc567ff51561c1a81a66f8608d90e6ea9d3: Status 404 returned error can't find the container with id 5b60db6f5c8d7b1271ed5b2417cd8dc567ff51561c1a81a66f8608d90e6ea9d3 Nov 02 23:20:02.164092 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:02.164058 5076 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 23:20:02.164274 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:02.164109 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" podUID="984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 23:20:02.353672 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:02.353641 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-027689-5864dbbf67-j588d" Nov 02 23:20:02.383631 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:02.383596 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7415-8368a1" event={"ID":"63ef8260-6155-4d11-a82c-bb48d232a4ac","Type":"ContainerStarted","Data":"5b60db6f5c8d7b1271ed5b2417cd8dc567ff51561c1a81a66f8608d90e6ea9d3"} Nov 02 23:20:03.164394 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:03.164356 5076 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 23:20:03.164868 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:03.164419 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" podUID="984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 23:20:04.164548 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:04.164033 5076 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 23:20:04.164548 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:04.164152 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" podUID="984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 23:20:05.164194 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:05.164113 5076 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 23:20:05.164194 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:05.164165 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" podUID="984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 23:20:05.405956 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:05.405913 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7415-8368a1" event={"ID":"63ef8260-6155-4d11-a82c-bb48d232a4ac","Type":"ContainerStarted","Data":"ee968d607805f786f0433b931805f1891d56364c4f12f7937e8ffb5a097794ed"} Nov 02 23:20:05.524260 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:05.515235 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7415-8368a1" podStartSLOduration=2.076293174 podStartE2EDuration="5.515216197s" totalImagesPullingTime="3.438923023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:20:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:20:05.51221527 +0000 UTC m=+1654.480747057" watchObservedRunningTime="2026-11-02 23:20:05.515216197 +0000 UTC m=+1654.483747984" Nov 02 23:20:06.168799 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:06.168765 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" Nov 02 23:20:06.169243 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:06.169223 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" Nov 02 23:20:06.169806 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:06.169791 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" Nov 02 23:20:06.958496 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:06.958463 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7415-8368a1"] Nov 02 23:20:06.965296 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:06.965261 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7415-8368a1"] Nov 02 23:20:07.424612 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:07.424514 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7415-8368a1" podUID="63ef8260-6155-4d11-a82c-bb48d232a4ac" containerName="test-curl-7415-8368a1" containerID="cri-o://ee968d607805f786f0433b931805f1891d56364c4f12f7937e8ffb5a097794ed" gracePeriod=2 Nov 02 23:20:09.678847 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:09.678787 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7415-8368a1" Nov 02 23:20:09.789211 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:09.789168 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63ef8260-6155-4d11-a82c-bb48d232a4ac-kube-api-access-p726w\" (UniqueName: \"kubernetes.io/projected/63ef8260-6155-4d11-a82c-bb48d232a4ac-kube-api-access-p726w\") pod \"63ef8260-6155-4d11-a82c-bb48d232a4ac\" (UID: \"63ef8260-6155-4d11-a82c-bb48d232a4ac\") " Nov 02 23:20:09.791524 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:09.791487 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63ef8260-6155-4d11-a82c-bb48d232a4ac-kube-api-access-p726w" pod "63ef8260-6155-4d11-a82c-bb48d232a4ac" (UID: "63ef8260-6155-4d11-a82c-bb48d232a4ac"). InnerVolumeSpecName "kube-api-access-p726w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:20:09.884733 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:20:09.884687 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:20:09.890692 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:09.890666 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p726w\" (UniqueName: \"kubernetes.io/projected/63ef8260-6155-4d11-a82c-bb48d232a4ac-kube-api-access-p726w\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:20:10.453744 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:10.453708 5076 generic.go:356] "Generic (PLEG): container finished" podID="63ef8260-6155-4d11-a82c-bb48d232a4ac" containerID="ee968d607805f786f0433b931805f1891d56364c4f12f7937e8ffb5a097794ed" exitCode=137 Nov 02 23:20:10.453961 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:10.453764 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7415-8368a1" Nov 02 23:20:10.453961 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:10.453804 5076 scope.go:122] "RemoveContainer" containerID="ee968d607805f786f0433b931805f1891d56364c4f12f7937e8ffb5a097794ed" Nov 02 23:20:10.464745 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:10.464730 5076 scope.go:122] "RemoveContainer" containerID="ee968d607805f786f0433b931805f1891d56364c4f12f7937e8ffb5a097794ed" Nov 02 23:20:10.465083 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:20:10.465063 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee968d607805f786f0433b931805f1891d56364c4f12f7937e8ffb5a097794ed\": container with ID starting with ee968d607805f786f0433b931805f1891d56364c4f12f7937e8ffb5a097794ed not found: ID does not exist" containerID="ee968d607805f786f0433b931805f1891d56364c4f12f7937e8ffb5a097794ed" Nov 02 23:20:10.465176 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:10.465091 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee968d607805f786f0433b931805f1891d56364c4f12f7937e8ffb5a097794ed"} err="failed to get container status \"ee968d607805f786f0433b931805f1891d56364c4f12f7937e8ffb5a097794ed\": rpc error: code = NotFound desc = could not find container \"ee968d607805f786f0433b931805f1891d56364c4f12f7937e8ffb5a097794ed\": container with ID starting with ee968d607805f786f0433b931805f1891d56364c4f12f7937e8ffb5a097794ed not found: ID does not exist" Nov 02 23:20:10.512532 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:20:10.512500 5076 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7415-8368a1\" is forbidden: User \"system:node:ip-10-0-130-211.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-211.us-west-2.compute.internal' and this object" podUID="63ef8260-6155-4d11-a82c-bb48d232a4ac" pod="opendatahub/test-curl-7415-8368a1" Nov 02 23:20:11.149832 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:11.149751 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63ef8260-6155-4d11-a82c-bb48d232a4ac" path="/var/lib/kubelet/pods/63ef8260-6155-4d11-a82c-bb48d232a4ac/volumes" Nov 02 23:20:11.193786 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:20:11.193739 5076 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7415-8368a1\" is forbidden: User \"system:node:ip-10-0-130-211.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-211.us-west-2.compute.internal' and this object" podUID="63ef8260-6155-4d11-a82c-bb48d232a4ac" pod="opendatahub/test-curl-7415-8368a1" Nov 02 23:20:18.540616 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:18.540580 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-818862-c954974c4-2vb2t"] Nov 02 23:20:18.541224 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:18.541209 5076 cpu_manager.go:414] "containerMap: removing container" podUID="63ef8260-6155-4d11-a82c-bb48d232a4ac" containerName="test-curl-7415-8368a1" Nov 02 23:20:18.541286 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:18.541226 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63ef8260-6155-4d11-a82c-bb48d232a4ac" containerName="test-curl-7415-8368a1" Nov 02 23:20:18.541376 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:18.541365 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="63ef8260-6155-4d11-a82c-bb48d232a4ac" containerName="test-curl-7415-8368a1" Nov 02 23:20:18.546314 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:18.546293 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-818862-c954974c4-2vb2t" Nov 02 23:20:18.598828 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:18.598796 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-818862-c954974c4-2vb2t"] Nov 02 23:20:18.610931 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:18.609531 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9whl\" (UniqueName: \"kubernetes.io/projected/1160858d-2d42-445c-81f9-644fe3a81e5f-kube-api-access-p9whl\") pod \"maas-api-e2e-api-b-818862-c954974c4-2vb2t\" (UID: \"1160858d-2d42-445c-81f9-644fe3a81e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-818862-c954974c4-2vb2t" Nov 02 23:20:18.610931 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:18.609657 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1160858d-2d42-445c-81f9-644fe3a81e5f-maas-api-tls\") pod \"maas-api-e2e-api-b-818862-c954974c4-2vb2t\" (UID: \"1160858d-2d42-445c-81f9-644fe3a81e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-818862-c954974c4-2vb2t" Nov 02 23:20:18.610931 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:18.609757 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1160858d-2d42-445c-81f9-644fe3a81e5f-metrics-certs\") pod \"maas-api-e2e-api-b-818862-c954974c4-2vb2t\" (UID: \"1160858d-2d42-445c-81f9-644fe3a81e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-818862-c954974c4-2vb2t" Nov 02 23:20:18.610931 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:18.609792 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1160858d-2d42-445c-81f9-644fe3a81e5f-openshift-service-ca\") pod \"maas-api-e2e-api-b-818862-c954974c4-2vb2t\" (UID: \"1160858d-2d42-445c-81f9-644fe3a81e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-818862-c954974c4-2vb2t" Nov 02 23:20:18.641705 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:18.641668 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-818862-serving-cert\"" Nov 02 23:20:18.711063 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:18.711019 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1160858d-2d42-445c-81f9-644fe3a81e5f-metrics-certs\") pod \"maas-api-e2e-api-b-818862-c954974c4-2vb2t\" (UID: \"1160858d-2d42-445c-81f9-644fe3a81e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-818862-c954974c4-2vb2t" Nov 02 23:20:18.711276 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:18.711095 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1160858d-2d42-445c-81f9-644fe3a81e5f-openshift-service-ca\") pod \"maas-api-e2e-api-b-818862-c954974c4-2vb2t\" (UID: \"1160858d-2d42-445c-81f9-644fe3a81e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-818862-c954974c4-2vb2t" Nov 02 23:20:18.711276 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:18.711250 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9whl\" (UniqueName: \"kubernetes.io/projected/1160858d-2d42-445c-81f9-644fe3a81e5f-kube-api-access-p9whl\") pod \"maas-api-e2e-api-b-818862-c954974c4-2vb2t\" (UID: \"1160858d-2d42-445c-81f9-644fe3a81e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-818862-c954974c4-2vb2t" Nov 02 23:20:18.711392 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:18.711342 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1160858d-2d42-445c-81f9-644fe3a81e5f-maas-api-tls\") pod \"maas-api-e2e-api-b-818862-c954974c4-2vb2t\" (UID: \"1160858d-2d42-445c-81f9-644fe3a81e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-818862-c954974c4-2vb2t" Nov 02 23:20:18.711494 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:20:18.711476 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-818862-serving-cert: secret "maas-api-e2e-api-b-818862-serving-cert" not found Nov 02 23:20:18.711585 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:20:18.711573 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1160858d-2d42-445c-81f9-644fe3a81e5f-maas-api-tls podName:1160858d-2d42-445c-81f9-644fe3a81e5f nodeName:}" failed. No retries permitted until 2026-11-02 23:20:19.211549355 +0000 UTC m=+1668.180081125 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1160858d-2d42-445c-81f9-644fe3a81e5f-maas-api-tls") pod "maas-api-e2e-api-b-818862-c954974c4-2vb2t" (UID: "1160858d-2d42-445c-81f9-644fe3a81e5f") : secret "maas-api-e2e-api-b-818862-serving-cert" not found Nov 02 23:20:18.712514 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:18.712483 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1160858d-2d42-445c-81f9-644fe3a81e5f-openshift-service-ca\") pod \"maas-api-e2e-api-b-818862-c954974c4-2vb2t\" (UID: \"1160858d-2d42-445c-81f9-644fe3a81e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-818862-c954974c4-2vb2t" Nov 02 23:20:18.714130 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:18.714109 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1160858d-2d42-445c-81f9-644fe3a81e5f-metrics-certs\") pod \"maas-api-e2e-api-b-818862-c954974c4-2vb2t\" (UID: \"1160858d-2d42-445c-81f9-644fe3a81e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-818862-c954974c4-2vb2t" Nov 02 23:20:18.768214 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:18.768185 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9whl\" (UniqueName: \"kubernetes.io/projected/1160858d-2d42-445c-81f9-644fe3a81e5f-kube-api-access-p9whl\") pod \"maas-api-e2e-api-b-818862-c954974c4-2vb2t\" (UID: \"1160858d-2d42-445c-81f9-644fe3a81e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-818862-c954974c4-2vb2t" Nov 02 23:20:19.225420 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:19.225372 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1160858d-2d42-445c-81f9-644fe3a81e5f-maas-api-tls\") pod \"maas-api-e2e-api-b-818862-c954974c4-2vb2t\" (UID: \"1160858d-2d42-445c-81f9-644fe3a81e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-818862-c954974c4-2vb2t" Nov 02 23:20:19.227845 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:19.227815 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1160858d-2d42-445c-81f9-644fe3a81e5f-maas-api-tls\") pod \"maas-api-e2e-api-b-818862-c954974c4-2vb2t\" (UID: \"1160858d-2d42-445c-81f9-644fe3a81e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-818862-c954974c4-2vb2t" Nov 02 23:20:19.460472 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:19.460435 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-818862-c954974c4-2vb2t" Nov 02 23:20:19.832414 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:19.832351 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-818862-c954974c4-2vb2t"] Nov 02 23:20:20.533723 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:20.533688 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-818862-c954974c4-2vb2t" event={"ID":"1160858d-2d42-445c-81f9-644fe3a81e5f","Type":"ContainerStarted","Data":"e4c52aee2251b01e8f84aa0df7ca20b1789aa47e308edb44b3c58988a3abcab3"} Nov 02 23:20:21.541374 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:21.541338 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-818862-c954974c4-2vb2t" event={"ID":"1160858d-2d42-445c-81f9-644fe3a81e5f","Type":"ContainerStarted","Data":"918cf39149830bbddc2ba2ea88a26301eb4d79f7892c3fa23d04ad289a3852bc"} Nov 02 23:20:21.541798 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:21.541476 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-818862-c954974c4-2vb2t" Nov 02 23:20:21.653884 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:21.653824 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-818862-c954974c4-2vb2t" podStartSLOduration=2.6067493710000003 podStartE2EDuration="3.653809995s" totalImagesPullingTime="1.047060624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:20:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:20:21.650830403 +0000 UTC m=+1670.619362197" watchObservedRunningTime="2026-11-02 23:20:21.653809995 +0000 UTC m=+1670.622341779" Nov 02 23:20:23.115621 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:23.115584 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ede388-667f44657f-5hr7f"] Nov 02 23:20:23.116069 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:23.115782 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ede388-667f44657f-5hr7f" podUID="b77a4e24-71dc-49c2-8832-f64c72e0e54a" containerName="maas-api" containerID="cri-o://13064b3020a8cb56457c9c729b47379a35bfb6e94176af55641e8ee0f8391938" gracePeriod=30 Nov 02 23:20:23.476280 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:23.476257 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ede388-667f44657f-5hr7f" Nov 02 23:20:23.591680 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:23.591610 5076 generic.go:356] "Generic (PLEG): container finished" podID="b77a4e24-71dc-49c2-8832-f64c72e0e54a" containerID="13064b3020a8cb56457c9c729b47379a35bfb6e94176af55641e8ee0f8391938" exitCode=0 Nov 02 23:20:23.592026 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:23.591997 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ede388-667f44657f-5hr7f" event={"ID":"b77a4e24-71dc-49c2-8832-f64c72e0e54a","Type":"ContainerDied","Data":"13064b3020a8cb56457c9c729b47379a35bfb6e94176af55641e8ee0f8391938"} Nov 02 23:20:23.592246 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:23.592226 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ede388-667f44657f-5hr7f" event={"ID":"b77a4e24-71dc-49c2-8832-f64c72e0e54a","Type":"ContainerDied","Data":"109d55f277d7dcb2062e4bb494f8d40b4336eb775558c3daf458cb0d97d74c88"} Nov 02 23:20:23.592372 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:23.592357 5076 scope.go:122] "RemoveContainer" containerID="13064b3020a8cb56457c9c729b47379a35bfb6e94176af55641e8ee0f8391938" Nov 02 23:20:23.595972 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:23.595695 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ede388-667f44657f-5hr7f" Nov 02 23:20:23.608637 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:23.608556 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b77a4e24-71dc-49c2-8832-f64c72e0e54a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b77a4e24-71dc-49c2-8832-f64c72e0e54a-openshift-service-ca\") pod \"b77a4e24-71dc-49c2-8832-f64c72e0e54a\" (UID: \"b77a4e24-71dc-49c2-8832-f64c72e0e54a\") " Nov 02 23:20:23.608937 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:23.608917 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b77a4e24-71dc-49c2-8832-f64c72e0e54a-kube-api-access-kgf5r\" (UniqueName: \"kubernetes.io/projected/b77a4e24-71dc-49c2-8832-f64c72e0e54a-kube-api-access-kgf5r\") pod \"b77a4e24-71dc-49c2-8832-f64c72e0e54a\" (UID: \"b77a4e24-71dc-49c2-8832-f64c72e0e54a\") " Nov 02 23:20:23.609108 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:23.609092 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b77a4e24-71dc-49c2-8832-f64c72e0e54a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b77a4e24-71dc-49c2-8832-f64c72e0e54a-maas-api-tls\") pod \"b77a4e24-71dc-49c2-8832-f64c72e0e54a\" (UID: \"b77a4e24-71dc-49c2-8832-f64c72e0e54a\") " Nov 02 23:20:23.609451 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:23.609431 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b77a4e24-71dc-49c2-8832-f64c72e0e54a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b77a4e24-71dc-49c2-8832-f64c72e0e54a-metrics-certs\") pod \"b77a4e24-71dc-49c2-8832-f64c72e0e54a\" (UID: \"b77a4e24-71dc-49c2-8832-f64c72e0e54a\") " Nov 02 23:20:23.612685 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:23.610090 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b77a4e24-71dc-49c2-8832-f64c72e0e54a-openshift-service-ca" pod "b77a4e24-71dc-49c2-8832-f64c72e0e54a" (UID: "b77a4e24-71dc-49c2-8832-f64c72e0e54a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:20:23.615004 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:23.614963 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b77a4e24-71dc-49c2-8832-f64c72e0e54a-kube-api-access-kgf5r" pod "b77a4e24-71dc-49c2-8832-f64c72e0e54a" (UID: "b77a4e24-71dc-49c2-8832-f64c72e0e54a"). InnerVolumeSpecName "kube-api-access-kgf5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:20:23.622711 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:23.620128 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b77a4e24-71dc-49c2-8832-f64c72e0e54a-maas-api-tls" pod "b77a4e24-71dc-49c2-8832-f64c72e0e54a" (UID: "b77a4e24-71dc-49c2-8832-f64c72e0e54a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:20:23.624081 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:23.624054 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b77a4e24-71dc-49c2-8832-f64c72e0e54a-metrics-certs" pod "b77a4e24-71dc-49c2-8832-f64c72e0e54a" (UID: "b77a4e24-71dc-49c2-8832-f64c72e0e54a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:20:23.640127 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:23.640105 5076 scope.go:122] "RemoveContainer" containerID="13064b3020a8cb56457c9c729b47379a35bfb6e94176af55641e8ee0f8391938" Nov 02 23:20:23.641505 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:20:23.641482 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13064b3020a8cb56457c9c729b47379a35bfb6e94176af55641e8ee0f8391938\": container with ID starting with 13064b3020a8cb56457c9c729b47379a35bfb6e94176af55641e8ee0f8391938 not found: ID does not exist" containerID="13064b3020a8cb56457c9c729b47379a35bfb6e94176af55641e8ee0f8391938" Nov 02 23:20:23.641594 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:23.641512 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13064b3020a8cb56457c9c729b47379a35bfb6e94176af55641e8ee0f8391938"} err="failed to get container status \"13064b3020a8cb56457c9c729b47379a35bfb6e94176af55641e8ee0f8391938\": rpc error: code = NotFound desc = could not find container \"13064b3020a8cb56457c9c729b47379a35bfb6e94176af55641e8ee0f8391938\": container with ID starting with 13064b3020a8cb56457c9c729b47379a35bfb6e94176af55641e8ee0f8391938 not found: ID does not exist" Nov 02 23:20:23.721964 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:23.721712 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b77a4e24-71dc-49c2-8832-f64c72e0e54a-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:20:23.721964 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:23.721759 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgf5r\" (UniqueName: \"kubernetes.io/projected/b77a4e24-71dc-49c2-8832-f64c72e0e54a-kube-api-access-kgf5r\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:20:23.721964 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:23.721777 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b77a4e24-71dc-49c2-8832-f64c72e0e54a-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:20:23.721964 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:23.721791 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b77a4e24-71dc-49c2-8832-f64c72e0e54a-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:20:24.094843 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:24.094751 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ede388-667f44657f-5hr7f"] Nov 02 23:20:24.160929 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:24.158032 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ede388-667f44657f-5hr7f"] Nov 02 23:20:25.160818 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:25.160784 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b77a4e24-71dc-49c2-8832-f64c72e0e54a" path="/var/lib/kubelet/pods/b77a4e24-71dc-49c2-8832-f64c72e0e54a/volumes" Nov 02 23:20:27.554279 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:27.554240 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-818862-c954974c4-2vb2t" Nov 02 23:20:33.465718 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:33.465679 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh"] Nov 02 23:20:33.466194 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:33.465871 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" podUID="818ebc94-d373-4c81-8c1f-d8a1a30bcbb1" containerName="istio-proxy" containerID="cri-o://323da69353697cbd4bc50143494f278662d9ef246af24a61c917d0e12f21411e" gracePeriod=30 Nov 02 23:20:34.750128 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:34.750053 5076 generic.go:356] "Generic (PLEG): container finished" podID="801851c2-20cb-4816-9a90-032a209b4933" containerID="b9fdc573e3d71a04eddb008d6f94ca045c652354c2f97f4f60d06081b0adf7ce" exitCode=0 Nov 02 23:20:34.750543 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:34.750121 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" event={"ID":"801851c2-20cb-4816-9a90-032a209b4933","Type":"ContainerDied","Data":"b9fdc573e3d71a04eddb008d6f94ca045c652354c2f97f4f60d06081b0adf7ce"} Nov 02 23:20:34.750543 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:34.750527 5076 scope.go:122] "RemoveContainer" containerID="b9fdc573e3d71a04eddb008d6f94ca045c652354c2f97f4f60d06081b0adf7ce" Nov 02 23:20:34.752098 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:34.752070 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:20:34.752187 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:34.752136 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:20:34.752187 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:34.752168 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:20:34.799277 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:34.799208 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8"] Nov 02 23:20:34.799952 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:34.799924 5076 cpu_manager.go:414] "containerMap: removing container" podUID="b77a4e24-71dc-49c2-8832-f64c72e0e54a" containerName="maas-api" Nov 02 23:20:34.799952 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:34.799948 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b77a4e24-71dc-49c2-8832-f64c72e0e54a" containerName="maas-api" Nov 02 23:20:34.800159 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:34.800119 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="b77a4e24-71dc-49c2-8832-f64c72e0e54a" containerName="maas-api" Nov 02 23:20:34.839260 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:34.839234 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8" Nov 02 23:20:34.856996 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:34.856958 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8"] Nov 02 23:20:34.889422 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:34.889398 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-a69f65-serving-cert\"" Nov 02 23:20:34.950783 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:34.950735 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11c1048d-56a0-4abf-97a0-f0b99ed82d7c-metrics-certs\") pod \"maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8\" (UID: \"11c1048d-56a0-4abf-97a0-f0b99ed82d7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8" Nov 02 23:20:34.950970 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:34.950802 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11c1048d-56a0-4abf-97a0-f0b99ed82d7c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8\" (UID: \"11c1048d-56a0-4abf-97a0-f0b99ed82d7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8" Nov 02 23:20:34.950970 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:34.950912 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtst8\" (UniqueName: \"kubernetes.io/projected/11c1048d-56a0-4abf-97a0-f0b99ed82d7c-kube-api-access-dtst8\") pod \"maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8\" (UID: \"11c1048d-56a0-4abf-97a0-f0b99ed82d7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8" Nov 02 23:20:34.951216 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:34.951191 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11c1048d-56a0-4abf-97a0-f0b99ed82d7c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8\" (UID: \"11c1048d-56a0-4abf-97a0-f0b99ed82d7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8" Nov 02 23:20:35.052726 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:35.052587 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11c1048d-56a0-4abf-97a0-f0b99ed82d7c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8\" (UID: \"11c1048d-56a0-4abf-97a0-f0b99ed82d7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8" Nov 02 23:20:35.052917 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:35.052773 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11c1048d-56a0-4abf-97a0-f0b99ed82d7c-metrics-certs\") pod \"maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8\" (UID: \"11c1048d-56a0-4abf-97a0-f0b99ed82d7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8" Nov 02 23:20:35.052917 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:35.052817 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11c1048d-56a0-4abf-97a0-f0b99ed82d7c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8\" (UID: \"11c1048d-56a0-4abf-97a0-f0b99ed82d7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8" Nov 02 23:20:35.052917 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:35.052851 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtst8\" (UniqueName: \"kubernetes.io/projected/11c1048d-56a0-4abf-97a0-f0b99ed82d7c-kube-api-access-dtst8\") pod \"maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8\" (UID: \"11c1048d-56a0-4abf-97a0-f0b99ed82d7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8" Nov 02 23:20:35.053091 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:20:35.053067 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-a69f65-serving-cert: secret "maas-api-e2e-ipp-a-a69f65-serving-cert" not found Nov 02 23:20:35.053151 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:20:35.053142 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11c1048d-56a0-4abf-97a0-f0b99ed82d7c-maas-api-tls podName:11c1048d-56a0-4abf-97a0-f0b99ed82d7c nodeName:}" failed. No retries permitted until 2026-11-02 23:20:35.553124321 +0000 UTC m=+1684.521656101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/11c1048d-56a0-4abf-97a0-f0b99ed82d7c-maas-api-tls") pod "maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8" (UID: "11c1048d-56a0-4abf-97a0-f0b99ed82d7c") : secret "maas-api-e2e-ipp-a-a69f65-serving-cert" not found Nov 02 23:20:35.054045 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:35.054021 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11c1048d-56a0-4abf-97a0-f0b99ed82d7c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8\" (UID: \"11c1048d-56a0-4abf-97a0-f0b99ed82d7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8" Nov 02 23:20:35.064766 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:35.064739 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11c1048d-56a0-4abf-97a0-f0b99ed82d7c-metrics-certs\") pod \"maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8\" (UID: \"11c1048d-56a0-4abf-97a0-f0b99ed82d7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8" Nov 02 23:20:35.107663 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:35.107631 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtst8\" (UniqueName: \"kubernetes.io/projected/11c1048d-56a0-4abf-97a0-f0b99ed82d7c-kube-api-access-dtst8\") pod \"maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8\" (UID: \"11c1048d-56a0-4abf-97a0-f0b99ed82d7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8" Nov 02 23:20:35.568366 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:35.568325 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11c1048d-56a0-4abf-97a0-f0b99ed82d7c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8\" (UID: \"11c1048d-56a0-4abf-97a0-f0b99ed82d7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8" Nov 02 23:20:35.570769 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:35.570742 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11c1048d-56a0-4abf-97a0-f0b99ed82d7c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8\" (UID: \"11c1048d-56a0-4abf-97a0-f0b99ed82d7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8" Nov 02 23:20:35.758969 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:35.758936 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" event={"ID":"801851c2-20cb-4816-9a90-032a209b4933","Type":"ContainerStarted","Data":"e682adfcca449a0b35361d4d070d3ab6e1120b12afa8e688c98d0a99dab77b1e"} Nov 02 23:20:35.767471 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:35.767450 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8" Nov 02 23:20:36.166511 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:36.166424 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8"] Nov 02 23:20:36.166644 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:20:36.166616 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c1048d_56a0_4abf_97a0_f0b99ed82d7c.slice/crio-94b37f5ae412f0f6caf74f830ce8c63b391331429894eacb2c21cece3ff362c2 WatchSource:0}: Error finding container 94b37f5ae412f0f6caf74f830ce8c63b391331429894eacb2c21cece3ff362c2: Status 404 returned error can't find the container with id 94b37f5ae412f0f6caf74f830ce8c63b391331429894eacb2c21cece3ff362c2 Nov 02 23:20:36.569363 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:36.569327 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" Nov 02 23:20:36.569544 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:36.569522 5076 patch_prober.go:29] interesting pod/e2e-ipp-a-027689-openshift-default-749945876c-n6p62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 02 23:20:36.569611 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:36.569570 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" podUID="801851c2-20cb-4816-9a90-032a209b4933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 02 23:20:36.768812 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:36.768773 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8" event={"ID":"11c1048d-56a0-4abf-97a0-f0b99ed82d7c","Type":"ContainerStarted","Data":"94b37f5ae412f0f6caf74f830ce8c63b391331429894eacb2c21cece3ff362c2"} Nov 02 23:20:37.569604 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:37.569572 5076 patch_prober.go:29] interesting pod/e2e-ipp-a-027689-openshift-default-749945876c-n6p62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 02 23:20:37.569785 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:37.569635 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" podUID="801851c2-20cb-4816-9a90-032a209b4933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 02 23:20:37.777855 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:37.777815 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8" event={"ID":"11c1048d-56a0-4abf-97a0-f0b99ed82d7c","Type":"ContainerStarted","Data":"09e2017428c0e542bd25bc2a2f06563ef13ea23600f38525af80fe0f2286b23b"} Nov 02 23:20:37.778289 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:37.777921 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8" Nov 02 23:20:37.896287 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:37.896176 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8" podStartSLOduration=3.013702864 podStartE2EDuration="3.896163233s" totalImagesPullingTime="882.460369ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:20:37.894801539 +0000 UTC m=+1686.863333322" watchObservedRunningTime="2026-11-02 23:20:37.896163233 +0000 UTC m=+1686.864695016" Nov 02 23:20:38.569417 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:38.569377 5076 patch_prober.go:29] interesting pod/e2e-ipp-a-027689-openshift-default-749945876c-n6p62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 02 23:20:38.569557 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:38.569439 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" podUID="801851c2-20cb-4816-9a90-032a209b4933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 02 23:20:38.789374 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:38.789338 5076 generic.go:356] "Generic (PLEG): container finished" podID="818ebc94-d373-4c81-8c1f-d8a1a30bcbb1" containerID="323da69353697cbd4bc50143494f278662d9ef246af24a61c917d0e12f21411e" exitCode=0 Nov 02 23:20:38.789737 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:38.789407 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" event={"ID":"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1","Type":"ContainerDied","Data":"323da69353697cbd4bc50143494f278662d9ef246af24a61c917d0e12f21411e"} Nov 02 23:20:38.789737 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:38.789498 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" event={"ID":"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1","Type":"ContainerDied","Data":"a04f5f8d22eca270fa6adca92f48ef8116346bc439db87e76863ecf7bbdf4a37"} Nov 02 23:20:38.789737 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:38.789517 5076 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a04f5f8d22eca270fa6adca92f48ef8116346bc439db87e76863ecf7bbdf4a37" Nov 02 23:20:38.789737 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:38.789531 5076 scope.go:122] "RemoveContainer" containerID="ea9410cc6a4527d30d9e68d51b4abef2975fd9e8243fc7b05c70951d7fbe499e" Nov 02 23:20:38.795678 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:38.795654 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" Nov 02 23:20:38.950265 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:38.950226 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-istio-token\" (UniqueName: \"kubernetes.io/projected/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-istio-token\") pod \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\" (UID: \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\") " Nov 02 23:20:38.950473 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:38.950283 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-istio-envoy\") pod \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\" (UID: \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\") " Nov 02 23:20:38.950473 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:38.950308 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-workload-certs\") pod \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\" (UID: \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\") " Nov 02 23:20:38.950473 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:38.950350 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-istiod-ca-cert\") pod \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\" (UID: \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\") " Nov 02 23:20:38.950473 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:38.950386 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-istio-data\") pod \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\" (UID: \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\") " Nov 02 23:20:38.950473 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:38.950475 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-istio-podinfo\") pod \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\" (UID: \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\") " Nov 02 23:20:38.950761 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:38.950568 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-workload-certs" pod "818ebc94-d373-4c81-8c1f-d8a1a30bcbb1" (UID: "818ebc94-d373-4c81-8c1f-d8a1a30bcbb1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:20:38.950761 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:38.950579 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-workload-socket\") pod \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\" (UID: \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\") " Nov 02 23:20:38.950761 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:38.950740 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-credential-socket\") pod \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\" (UID: \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\") " Nov 02 23:20:38.950949 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:38.950756 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-istio-data" pod "818ebc94-d373-4c81-8c1f-d8a1a30bcbb1" (UID: "818ebc94-d373-4c81-8c1f-d8a1a30bcbb1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:20:38.950949 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:38.950773 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-workload-socket" pod "818ebc94-d373-4c81-8c1f-d8a1a30bcbb1" (UID: "818ebc94-d373-4c81-8c1f-d8a1a30bcbb1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:20:38.950949 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:38.950796 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-istiod-ca-cert" pod "818ebc94-d373-4c81-8c1f-d8a1a30bcbb1" (UID: "818ebc94-d373-4c81-8c1f-d8a1a30bcbb1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:20:38.951116 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:38.951029 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-credential-socket" pod "818ebc94-d373-4c81-8c1f-d8a1a30bcbb1" (UID: "818ebc94-d373-4c81-8c1f-d8a1a30bcbb1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:20:38.951175 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:38.951117 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-kube-api-access-kk4db\" (UniqueName: \"kubernetes.io/projected/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-kube-api-access-kk4db\") pod \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\" (UID: \"818ebc94-d373-4c81-8c1f-d8a1a30bcbb1\") " Nov 02 23:20:38.951688 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:38.951652 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:20:38.951688 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:38.951681 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:20:38.951864 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:38.951696 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:20:38.951864 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:38.951710 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:20:38.951864 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:38.951722 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:20:38.952692 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:38.952664 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-istio-envoy" pod "818ebc94-d373-4c81-8c1f-d8a1a30bcbb1" (UID: "818ebc94-d373-4c81-8c1f-d8a1a30bcbb1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:20:38.952692 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:38.952685 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-istio-podinfo" pod "818ebc94-d373-4c81-8c1f-d8a1a30bcbb1" (UID: "818ebc94-d373-4c81-8c1f-d8a1a30bcbb1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:20:38.952849 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:38.952670 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-istio-token" pod "818ebc94-d373-4c81-8c1f-d8a1a30bcbb1" (UID: "818ebc94-d373-4c81-8c1f-d8a1a30bcbb1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:20:38.953060 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:38.953042 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-kube-api-access-kk4db" pod "818ebc94-d373-4c81-8c1f-d8a1a30bcbb1" (UID: "818ebc94-d373-4c81-8c1f-d8a1a30bcbb1"). InnerVolumeSpecName "kube-api-access-kk4db". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:20:39.052566 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:39.052484 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:20:39.052566 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:39.052513 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk4db\" (UniqueName: \"kubernetes.io/projected/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-kube-api-access-kk4db\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:20:39.052566 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:39.052524 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:20:39.052566 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:39.052534 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:20:39.569855 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:39.569823 5076 patch_prober.go:29] interesting pod/e2e-ipp-a-027689-openshift-default-749945876c-n6p62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 02 23:20:39.570077 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:39.569883 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" podUID="801851c2-20cb-4816-9a90-032a209b4933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 02 23:20:39.876469 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:39.875635 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh" Nov 02 23:20:39.938705 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:20:39.938108 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:20:40.161115 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:40.161016 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh"] Nov 02 23:20:40.164221 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:40.164193 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ede388-openshift-default-7b95c8c6ff-9mhrh"] Nov 02 23:20:40.569790 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:40.569749 5076 patch_prober.go:29] interesting pod/e2e-ipp-a-027689-openshift-default-749945876c-n6p62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 02 23:20:40.570033 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:40.569811 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" podUID="801851c2-20cb-4816-9a90-032a209b4933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 02 23:20:41.148944 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:41.148907 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="818ebc94-d373-4c81-8c1f-d8a1a30bcbb1" path="/var/lib/kubelet/pods/818ebc94-d373-4c81-8c1f-d8a1a30bcbb1/volumes" Nov 02 23:20:41.574682 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:41.574650 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" Nov 02 23:20:41.575175 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:41.575155 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" Nov 02 23:20:41.575939 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:41.575920 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" Nov 02 23:20:43.795932 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:43.795875 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8" Nov 02 23:20:55.022080 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:55.022046 5076 generic.go:356] "Generic (PLEG): container finished" podID="9425db69-12a7-480d-b9a5-ef92b281a8ff" containerID="701de5b7736526c41ee427ab87c782202f12e81d4468487a14a205faa9b5ebe0" exitCode=0 Nov 02 23:20:55.022532 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:55.022121 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" event={"ID":"9425db69-12a7-480d-b9a5-ef92b281a8ff","Type":"ContainerDied","Data":"701de5b7736526c41ee427ab87c782202f12e81d4468487a14a205faa9b5ebe0"} Nov 02 23:20:55.022589 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:55.022567 5076 scope.go:122] "RemoveContainer" containerID="701de5b7736526c41ee427ab87c782202f12e81d4468487a14a205faa9b5ebe0" Nov 02 23:20:55.024133 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:55.024093 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:20:55.024255 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:55.024187 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:20:55.024255 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:55.024237 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:20:56.041673 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:56.041634 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" event={"ID":"9425db69-12a7-480d-b9a5-ef92b281a8ff","Type":"ContainerStarted","Data":"94c9aac842b981b9662952e1f36ac69f84e703cf320bdb80a877f5777c999110"} Nov 02 23:20:56.122752 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:56.122302 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" Nov 02 23:20:56.122752 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:56.122437 5076 patch_prober.go:29] interesting pod/e2e-api-b-818862-openshift-default-667cf76c7-v8946 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 02 23:20:56.122752 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:56.122481 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" podUID="9425db69-12a7-480d-b9a5-ef92b281a8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 02 23:20:57.131913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:57.131262 5076 patch_prober.go:29] interesting pod/e2e-api-b-818862-openshift-default-667cf76c7-v8946 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 02 23:20:57.131913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:57.131321 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" podUID="9425db69-12a7-480d-b9a5-ef92b281a8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 02 23:20:58.117352 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:58.117311 5076 patch_prober.go:29] interesting pod/e2e-api-b-818862-openshift-default-667cf76c7-v8946 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 02 23:20:58.117634 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:58.117604 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" podUID="9425db69-12a7-480d-b9a5-ef92b281a8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 02 23:20:59.133918 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:59.132233 5076 patch_prober.go:29] interesting pod/e2e-api-b-818862-openshift-default-667cf76c7-v8946 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 02 23:20:59.133918 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:20:59.132309 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" podUID="9425db69-12a7-480d-b9a5-ef92b281a8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 02 23:21:00.117916 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:00.113278 5076 patch_prober.go:29] interesting pod/e2e-api-b-818862-openshift-default-667cf76c7-v8946 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 02 23:21:00.117916 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:00.113340 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" podUID="9425db69-12a7-480d-b9a5-ef92b281a8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 02 23:21:01.119969 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:01.119928 5076 patch_prober.go:29] interesting pod/e2e-api-b-818862-openshift-default-667cf76c7-v8946 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 02 23:21:01.120483 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:01.119986 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" podUID="9425db69-12a7-480d-b9a5-ef92b281a8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 02 23:21:01.847850 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:01.847812 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-027689-5b9556b9bf-5kzrd"] Nov 02 23:21:01.948982 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:01.948946 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-f647578d7-4jc4f"] Nov 02 23:21:02.115823 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:02.115749 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" Nov 02 23:21:02.116113 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:02.116081 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" Nov 02 23:21:02.116811 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:02.116791 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" Nov 02 23:21:06.848336 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:06.848290 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-027689-5b9556b9bf-5kzrd" podUID="77f3b39d-cd9c-46ae-86ee-83274a0d6d6c" containerName="payload-processing" containerID="cri-o://1a7c550bf4772935b692ef2543bf0e95dfa6a3dac843d4c1aa63f465e4c51cb3" gracePeriod=25 Nov 02 23:21:06.949754 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:06.949716 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-f647578d7-4jc4f" podUID="dd2370ae-3618-4464-939d-1821c4af19bd" containerName="payload-pre-processing" containerID="cri-o://bd99cdae79adce5ad2c830ed4a2fec0a7115eb9ce7a51285f7ab29cfbd579b0f" gracePeriod=25 Nov 02 23:21:07.114617 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:07.114588 5076 generic.go:356] "Generic (PLEG): container finished" podID="77f3b39d-cd9c-46ae-86ee-83274a0d6d6c" containerID="1a7c550bf4772935b692ef2543bf0e95dfa6a3dac843d4c1aa63f465e4c51cb3" exitCode=0 Nov 02 23:21:07.114763 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:07.114658 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-027689-5b9556b9bf-5kzrd" event={"ID":"77f3b39d-cd9c-46ae-86ee-83274a0d6d6c","Type":"ContainerDied","Data":"1a7c550bf4772935b692ef2543bf0e95dfa6a3dac843d4c1aa63f465e4c51cb3"} Nov 02 23:21:07.116480 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:07.116459 5076 generic.go:356] "Generic (PLEG): container finished" podID="dd2370ae-3618-4464-939d-1821c4af19bd" containerID="bd99cdae79adce5ad2c830ed4a2fec0a7115eb9ce7a51285f7ab29cfbd579b0f" exitCode=0 Nov 02 23:21:07.116603 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:07.116529 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-f647578d7-4jc4f" event={"ID":"dd2370ae-3618-4464-939d-1821c4af19bd","Type":"ContainerDied","Data":"bd99cdae79adce5ad2c830ed4a2fec0a7115eb9ce7a51285f7ab29cfbd579b0f"} Nov 02 23:21:07.151548 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:07.151517 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-027689-5b9556b9bf-5kzrd" Nov 02 23:21:07.208377 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:07.208345 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77f3b39d-cd9c-46ae-86ee-83274a0d6d6c-kube-api-access-8wdrs\" (UniqueName: \"kubernetes.io/projected/77f3b39d-cd9c-46ae-86ee-83274a0d6d6c-kube-api-access-8wdrs\") pod \"77f3b39d-cd9c-46ae-86ee-83274a0d6d6c\" (UID: \"77f3b39d-cd9c-46ae-86ee-83274a0d6d6c\") " Nov 02 23:21:07.208556 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:07.208527 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77f3b39d-cd9c-46ae-86ee-83274a0d6d6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f3b39d-cd9c-46ae-86ee-83274a0d6d6c-plugins-config-volume\") pod \"77f3b39d-cd9c-46ae-86ee-83274a0d6d6c\" (UID: \"77f3b39d-cd9c-46ae-86ee-83274a0d6d6c\") " Nov 02 23:21:07.209042 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:07.209009 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77f3b39d-cd9c-46ae-86ee-83274a0d6d6c-plugins-config-volume" pod "77f3b39d-cd9c-46ae-86ee-83274a0d6d6c" (UID: "77f3b39d-cd9c-46ae-86ee-83274a0d6d6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:21:07.209290 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:07.209269 5076 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f3b39d-cd9c-46ae-86ee-83274a0d6d6c-plugins-config-volume\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:21:07.212117 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:07.212084 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77f3b39d-cd9c-46ae-86ee-83274a0d6d6c-kube-api-access-8wdrs" pod "77f3b39d-cd9c-46ae-86ee-83274a0d6d6c" (UID: "77f3b39d-cd9c-46ae-86ee-83274a0d6d6c"). InnerVolumeSpecName "kube-api-access-8wdrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:21:07.232288 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:07.232266 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-f647578d7-4jc4f" Nov 02 23:21:07.310227 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:07.310196 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd2370ae-3618-4464-939d-1821c4af19bd-kube-api-access-whxbj\" (UniqueName: \"kubernetes.io/projected/dd2370ae-3618-4464-939d-1821c4af19bd-kube-api-access-whxbj\") pod \"dd2370ae-3618-4464-939d-1821c4af19bd\" (UID: \"dd2370ae-3618-4464-939d-1821c4af19bd\") " Nov 02 23:21:07.310376 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:07.310317 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd2370ae-3618-4464-939d-1821c4af19bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2370ae-3618-4464-939d-1821c4af19bd-plugins-config-volume\") pod \"dd2370ae-3618-4464-939d-1821c4af19bd\" (UID: \"dd2370ae-3618-4464-939d-1821c4af19bd\") " Nov 02 23:21:07.310749 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:07.310719 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd2370ae-3618-4464-939d-1821c4af19bd-plugins-config-volume" pod "dd2370ae-3618-4464-939d-1821c4af19bd" (UID: "dd2370ae-3618-4464-939d-1821c4af19bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:21:07.310951 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:07.310935 5076 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2370ae-3618-4464-939d-1821c4af19bd-plugins-config-volume\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:21:07.311005 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:07.310954 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wdrs\" (UniqueName: \"kubernetes.io/projected/77f3b39d-cd9c-46ae-86ee-83274a0d6d6c-kube-api-access-8wdrs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:21:07.312376 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:07.312354 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd2370ae-3618-4464-939d-1821c4af19bd-kube-api-access-whxbj" pod "dd2370ae-3618-4464-939d-1821c4af19bd" (UID: "dd2370ae-3618-4464-939d-1821c4af19bd"). InnerVolumeSpecName "kube-api-access-whxbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:21:07.412670 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:07.412576 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whxbj\" (UniqueName: \"kubernetes.io/projected/dd2370ae-3618-4464-939d-1821c4af19bd-kube-api-access-whxbj\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:21:08.124186 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:08.124104 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-f647578d7-4jc4f" Nov 02 23:21:08.124186 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:08.124139 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-f647578d7-4jc4f" event={"ID":"dd2370ae-3618-4464-939d-1821c4af19bd","Type":"ContainerDied","Data":"5ee37e24169776af0c850e4a43e1de56cd2dd964276140b177a42e3b94fb77fd"} Nov 02 23:21:08.124729 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:08.124191 5076 scope.go:122] "RemoveContainer" containerID="bd99cdae79adce5ad2c830ed4a2fec0a7115eb9ce7a51285f7ab29cfbd579b0f" Nov 02 23:21:08.125741 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:08.125723 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-027689-5b9556b9bf-5kzrd" Nov 02 23:21:08.125835 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:08.125735 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-027689-5b9556b9bf-5kzrd" event={"ID":"77f3b39d-cd9c-46ae-86ee-83274a0d6d6c","Type":"ContainerDied","Data":"d7813f797e32c0c7cf1bd70c6376d13ff46a6d1fa972a3dff4e623e828b65e70"} Nov 02 23:21:08.136069 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:08.136049 5076 scope.go:122] "RemoveContainer" containerID="1a7c550bf4772935b692ef2543bf0e95dfa6a3dac843d4c1aa63f465e4c51cb3" Nov 02 23:21:08.281430 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:08.281394 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-027689-5b9556b9bf-5kzrd"] Nov 02 23:21:08.328545 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:08.328507 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-027689-5b9556b9bf-5kzrd"] Nov 02 23:21:08.389717 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:08.389624 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-f647578d7-4jc4f"] Nov 02 23:21:08.392601 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:08.392577 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-f647578d7-4jc4f"] Nov 02 23:21:09.148901 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:09.148856 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77f3b39d-cd9c-46ae-86ee-83274a0d6d6c" path="/var/lib/kubelet/pods/77f3b39d-cd9c-46ae-86ee-83274a0d6d6c/volumes" Nov 02 23:21:09.149334 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:09.149318 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd2370ae-3618-4464-939d-1821c4af19bd" path="/var/lib/kubelet/pods/dd2370ae-3618-4464-939d-1821c4af19bd/volumes" Nov 02 23:21:09.886315 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:21:09.886276 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:21:21.214812 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:21.214779 5076 generic.go:356] "Generic (PLEG): container finished" podID="dd184b6f-3901-413f-ad6a-c7271183796f" containerID="cd88cb2e172b9fbd8f8043805751be2d839b447e7726bcc933dbecb51fbb208e" exitCode=0 Nov 02 23:21:21.215385 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:21.214836 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" event={"ID":"dd184b6f-3901-413f-ad6a-c7271183796f","Type":"ContainerDied","Data":"cd88cb2e172b9fbd8f8043805751be2d839b447e7726bcc933dbecb51fbb208e"} Nov 02 23:21:21.215385 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:21.215375 5076 scope.go:122] "RemoveContainer" containerID="cd88cb2e172b9fbd8f8043805751be2d839b447e7726bcc933dbecb51fbb208e" Nov 02 23:21:21.216980 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:21.216948 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:21:21.217070 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:21.217049 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:21:21.217106 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:21.217089 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:21:22.137150 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:22.134110 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:21:22.238288 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:22.238191 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" event={"ID":"dd184b6f-3901-413f-ad6a-c7271183796f","Type":"ContainerStarted","Data":"892a4ab283b16a8a62d8b6ad337a6d06a446b89f94647fd4c89beedca60ff4fe"} Nov 02 23:21:22.433972 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:22.433364 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:21:22.433972 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:22.433529 5076 patch_prober.go:29] interesting pod/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 02 23:21:22.433972 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:22.433572 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" podUID="dd184b6f-3901-413f-ad6a-c7271183796f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 02 23:21:23.434004 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:23.430294 5076 patch_prober.go:29] interesting pod/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 02 23:21:23.434004 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:23.430390 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" podUID="dd184b6f-3901-413f-ad6a-c7271183796f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 02 23:21:24.424350 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:24.424308 5076 patch_prober.go:29] interesting pod/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 02 23:21:24.424689 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:24.424659 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" podUID="dd184b6f-3901-413f-ad6a-c7271183796f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 02 23:21:25.428915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:25.426266 5076 patch_prober.go:29] interesting pod/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 02 23:21:25.428915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:25.426324 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" podUID="dd184b6f-3901-413f-ad6a-c7271183796f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 02 23:21:26.428915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:26.426347 5076 patch_prober.go:29] interesting pod/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 02 23:21:26.428915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:26.426406 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" podUID="dd184b6f-3901-413f-ad6a-c7271183796f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 02 23:21:27.428343 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:27.428300 5076 patch_prober.go:29] interesting pod/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 02 23:21:27.430781 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:27.430747 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" podUID="dd184b6f-3901-413f-ad6a-c7271183796f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 02 23:21:28.427468 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:28.427438 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:21:28.428015 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:28.427996 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:21:28.428731 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:28.428708 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:21:37.070097 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:37.070060 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-79385a-6d75566cc6-lqscm"] Nov 02 23:21:37.070575 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:37.070354 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79385a-6d75566cc6-lqscm" podUID="0a9b19ee-8d1b-4629-8a24-f59c6ca897dc" containerName="maas-api" containerID="cri-o://b09ffb6b37ce35f6eedfe9db80088c2d0d7f8c3352d70a1882c88320c5be58d2" gracePeriod=30 Nov 02 23:21:37.341868 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:37.341776 5076 generic.go:356] "Generic (PLEG): container finished" podID="0a9b19ee-8d1b-4629-8a24-f59c6ca897dc" containerID="b09ffb6b37ce35f6eedfe9db80088c2d0d7f8c3352d70a1882c88320c5be58d2" exitCode=0 Nov 02 23:21:37.342075 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:37.341983 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79385a-6d75566cc6-lqscm" event={"ID":"0a9b19ee-8d1b-4629-8a24-f59c6ca897dc","Type":"ContainerDied","Data":"b09ffb6b37ce35f6eedfe9db80088c2d0d7f8c3352d70a1882c88320c5be58d2"} Nov 02 23:21:37.674709 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:37.674622 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79385a-6d75566cc6-lqscm" Nov 02 23:21:37.809610 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:37.809572 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a9b19ee-8d1b-4629-8a24-f59c6ca897dc-kube-api-access-rw58g\" (UniqueName: \"kubernetes.io/projected/0a9b19ee-8d1b-4629-8a24-f59c6ca897dc-kube-api-access-rw58g\") pod \"0a9b19ee-8d1b-4629-8a24-f59c6ca897dc\" (UID: \"0a9b19ee-8d1b-4629-8a24-f59c6ca897dc\") " Nov 02 23:21:37.809812 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:37.809644 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a9b19ee-8d1b-4629-8a24-f59c6ca897dc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a9b19ee-8d1b-4629-8a24-f59c6ca897dc-metrics-certs\") pod \"0a9b19ee-8d1b-4629-8a24-f59c6ca897dc\" (UID: \"0a9b19ee-8d1b-4629-8a24-f59c6ca897dc\") " Nov 02 23:21:37.809812 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:37.809789 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a9b19ee-8d1b-4629-8a24-f59c6ca897dc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a9b19ee-8d1b-4629-8a24-f59c6ca897dc-maas-api-tls\") pod \"0a9b19ee-8d1b-4629-8a24-f59c6ca897dc\" (UID: \"0a9b19ee-8d1b-4629-8a24-f59c6ca897dc\") " Nov 02 23:21:37.809982 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:37.809838 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a9b19ee-8d1b-4629-8a24-f59c6ca897dc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9b19ee-8d1b-4629-8a24-f59c6ca897dc-openshift-service-ca\") pod \"0a9b19ee-8d1b-4629-8a24-f59c6ca897dc\" (UID: \"0a9b19ee-8d1b-4629-8a24-f59c6ca897dc\") " Nov 02 23:21:37.810480 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:37.810450 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a9b19ee-8d1b-4629-8a24-f59c6ca897dc-openshift-service-ca" pod "0a9b19ee-8d1b-4629-8a24-f59c6ca897dc" (UID: "0a9b19ee-8d1b-4629-8a24-f59c6ca897dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:21:37.812550 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:37.812525 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a9b19ee-8d1b-4629-8a24-f59c6ca897dc-maas-api-tls" pod "0a9b19ee-8d1b-4629-8a24-f59c6ca897dc" (UID: "0a9b19ee-8d1b-4629-8a24-f59c6ca897dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:21:37.813087 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:37.813047 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9b19ee-8d1b-4629-8a24-f59c6ca897dc-kube-api-access-rw58g" pod "0a9b19ee-8d1b-4629-8a24-f59c6ca897dc" (UID: "0a9b19ee-8d1b-4629-8a24-f59c6ca897dc"). InnerVolumeSpecName "kube-api-access-rw58g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:21:37.814252 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:37.814224 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a9b19ee-8d1b-4629-8a24-f59c6ca897dc-metrics-certs" pod "0a9b19ee-8d1b-4629-8a24-f59c6ca897dc" (UID: "0a9b19ee-8d1b-4629-8a24-f59c6ca897dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:21:37.912525 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:37.912491 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw58g\" (UniqueName: \"kubernetes.io/projected/0a9b19ee-8d1b-4629-8a24-f59c6ca897dc-kube-api-access-rw58g\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:21:37.912525 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:37.912521 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a9b19ee-8d1b-4629-8a24-f59c6ca897dc-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:21:37.912525 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:37.912531 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a9b19ee-8d1b-4629-8a24-f59c6ca897dc-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:21:37.912790 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:37.912541 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9b19ee-8d1b-4629-8a24-f59c6ca897dc-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:21:38.367169 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:38.367129 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79385a-6d75566cc6-lqscm" Nov 02 23:21:38.367169 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:38.367159 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79385a-6d75566cc6-lqscm" event={"ID":"0a9b19ee-8d1b-4629-8a24-f59c6ca897dc","Type":"ContainerDied","Data":"12140ee5e5f9cc72635a698ee2d2e1147ebb8980b14f8f5d12fc45e1581fd2fb"} Nov 02 23:21:38.367768 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:38.367232 5076 scope.go:122] "RemoveContainer" containerID="b09ffb6b37ce35f6eedfe9db80088c2d0d7f8c3352d70a1882c88320c5be58d2" Nov 02 23:21:38.566665 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:38.549967 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-79385a-6d75566cc6-lqscm"] Nov 02 23:21:38.591921 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:38.591673 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-79385a-6d75566cc6-lqscm"] Nov 02 23:21:39.155548 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:39.155505 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a9b19ee-8d1b-4629-8a24-f59c6ca897dc" path="/var/lib/kubelet/pods/0a9b19ee-8d1b-4629-8a24-f59c6ca897dc/volumes" Nov 02 23:21:39.885015 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:21:39.884980 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:21:47.972807 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:47.972770 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn"] Nov 02 23:21:47.976672 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:47.976645 5076 cpu_manager.go:414] "containerMap: removing container" podUID="dd2370ae-3618-4464-939d-1821c4af19bd" containerName="payload-pre-processing" Nov 02 23:21:47.977334 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:47.977312 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd2370ae-3618-4464-939d-1821c4af19bd" containerName="payload-pre-processing" Nov 02 23:21:47.977566 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:47.977553 5076 cpu_manager.go:414] "containerMap: removing container" podUID="818ebc94-d373-4c81-8c1f-d8a1a30bcbb1" containerName="istio-proxy" Nov 02 23:21:47.977682 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:47.977670 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="818ebc94-d373-4c81-8c1f-d8a1a30bcbb1" containerName="istio-proxy" Nov 02 23:21:47.977782 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:47.977773 5076 cpu_manager.go:414] "containerMap: removing container" podUID="77f3b39d-cd9c-46ae-86ee-83274a0d6d6c" containerName="payload-processing" Nov 02 23:21:47.977925 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:47.977913 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77f3b39d-cd9c-46ae-86ee-83274a0d6d6c" containerName="payload-processing" Nov 02 23:21:47.978070 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:47.978057 5076 cpu_manager.go:414] "containerMap: removing container" podUID="818ebc94-d373-4c81-8c1f-d8a1a30bcbb1" containerName="istio-proxy" Nov 02 23:21:47.978149 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:47.978139 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="818ebc94-d373-4c81-8c1f-d8a1a30bcbb1" containerName="istio-proxy" Nov 02 23:21:47.978266 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:47.978256 5076 cpu_manager.go:414] "containerMap: removing container" podUID="0a9b19ee-8d1b-4629-8a24-f59c6ca897dc" containerName="maas-api" Nov 02 23:21:47.978337 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:47.978328 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a9b19ee-8d1b-4629-8a24-f59c6ca897dc" containerName="maas-api" Nov 02 23:21:47.978580 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:47.978566 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="818ebc94-d373-4c81-8c1f-d8a1a30bcbb1" containerName="istio-proxy" Nov 02 23:21:47.978676 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:47.978666 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="77f3b39d-cd9c-46ae-86ee-83274a0d6d6c" containerName="payload-processing" Nov 02 23:21:47.978758 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:47.978749 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="818ebc94-d373-4c81-8c1f-d8a1a30bcbb1" containerName="istio-proxy" Nov 02 23:21:47.978838 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:47.978830 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a9b19ee-8d1b-4629-8a24-f59c6ca897dc" containerName="maas-api" Nov 02 23:21:47.978937 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:47.978927 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd2370ae-3618-4464-939d-1821c4af19bd" containerName="payload-pre-processing" Nov 02 23:21:47.982285 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:47.982269 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" Nov 02 23:21:48.099751 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:48.099704 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1632e0b-a576-4db6-ba82-dc11e89a7c00-workload-socket\") pod \"e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn\" (UID: \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\") " pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" Nov 02 23:21:48.100011 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:48.099990 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1632e0b-a576-4db6-ba82-dc11e89a7c00-istio-token\") pod \"e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn\" (UID: \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\") " pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" Nov 02 23:21:48.100280 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:48.100253 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1632e0b-a576-4db6-ba82-dc11e89a7c00-istio-data\") pod \"e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn\" (UID: \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\") " pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" Nov 02 23:21:48.100417 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:48.100403 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvmhq\" (UniqueName: \"kubernetes.io/projected/a1632e0b-a576-4db6-ba82-dc11e89a7c00-kube-api-access-hvmhq\") pod \"e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn\" (UID: \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\") " pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" Nov 02 23:21:48.100546 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:48.100530 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1632e0b-a576-4db6-ba82-dc11e89a7c00-workload-certs\") pod \"e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn\" (UID: \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\") " pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" Nov 02 23:21:48.103970 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:48.103938 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1632e0b-a576-4db6-ba82-dc11e89a7c00-credential-socket\") pod \"e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn\" (UID: \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\") " pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" Nov 02 23:21:48.107987 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:48.107953 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1632e0b-a576-4db6-ba82-dc11e89a7c00-istio-envoy\") pod \"e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn\" (UID: \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\") " pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" Nov 02 23:21:48.117013 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:48.116975 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1632e0b-a576-4db6-ba82-dc11e89a7c00-istiod-ca-cert\") pod \"e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn\" (UID: \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\") " pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" Nov 02 23:21:48.117225 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:48.117208 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1632e0b-a576-4db6-ba82-dc11e89a7c00-istio-podinfo\") pod \"e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn\" (UID: \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\") " pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" Nov 02 23:21:48.160949 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:48.153068 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-6a894c-openshift-default-dockercfg-rs5rf\"" Nov 02 23:21:48.171802 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:48.171761 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn"] Nov 02 23:21:48.221197 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:48.221159 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1632e0b-a576-4db6-ba82-dc11e89a7c00-istio-data\") pod \"e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn\" (UID: \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\") " pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" Nov 02 23:21:48.221637 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:48.221605 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvmhq\" (UniqueName: \"kubernetes.io/projected/a1632e0b-a576-4db6-ba82-dc11e89a7c00-kube-api-access-hvmhq\") pod \"e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn\" (UID: \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\") " pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" Nov 02 23:21:48.221793 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:48.221777 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1632e0b-a576-4db6-ba82-dc11e89a7c00-workload-certs\") pod \"e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn\" (UID: \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\") " pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" Nov 02 23:21:48.222031 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:48.222010 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1632e0b-a576-4db6-ba82-dc11e89a7c00-credential-socket\") pod \"e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn\" (UID: \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\") " pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" Nov 02 23:21:48.223467 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:48.223407 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1632e0b-a576-4db6-ba82-dc11e89a7c00-istio-envoy\") pod \"e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn\" (UID: \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\") " pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" Nov 02 23:21:48.224023 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:48.224004 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1632e0b-a576-4db6-ba82-dc11e89a7c00-istiod-ca-cert\") pod \"e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn\" (UID: \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\") " pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" Nov 02 23:21:48.225205 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:48.225178 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1632e0b-a576-4db6-ba82-dc11e89a7c00-istio-podinfo\") pod \"e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn\" (UID: \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\") " pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" Nov 02 23:21:48.225790 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:48.225769 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1632e0b-a576-4db6-ba82-dc11e89a7c00-workload-socket\") pod \"e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn\" (UID: \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\") " pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" Nov 02 23:21:48.226254 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:48.226237 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1632e0b-a576-4db6-ba82-dc11e89a7c00-istio-token\") pod \"e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn\" (UID: \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\") " pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" Nov 02 23:21:48.227218 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:48.225109 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1632e0b-a576-4db6-ba82-dc11e89a7c00-istiod-ca-cert\") pod \"e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn\" (UID: \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\") " pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" Nov 02 23:21:48.227357 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:48.223293 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1632e0b-a576-4db6-ba82-dc11e89a7c00-credential-socket\") pod \"e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn\" (UID: \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\") " pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" Nov 02 23:21:48.227462 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:48.226171 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1632e0b-a576-4db6-ba82-dc11e89a7c00-workload-socket\") pod \"e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn\" (UID: \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\") " pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" Nov 02 23:21:48.227564 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:48.222810 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1632e0b-a576-4db6-ba82-dc11e89a7c00-istio-data\") pod \"e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn\" (UID: \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\") " pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" Nov 02 23:21:48.227662 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:48.223073 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1632e0b-a576-4db6-ba82-dc11e89a7c00-workload-certs\") pod \"e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn\" (UID: \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\") " pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" Nov 02 23:21:48.306505 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:48.306401 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1632e0b-a576-4db6-ba82-dc11e89a7c00-istio-envoy\") pod \"e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn\" (UID: \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\") " pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" Nov 02 23:21:48.307454 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:48.307429 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1632e0b-a576-4db6-ba82-dc11e89a7c00-istio-podinfo\") pod \"e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn\" (UID: \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\") " pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" Nov 02 23:21:48.310065 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:48.310040 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvmhq\" (UniqueName: \"kubernetes.io/projected/a1632e0b-a576-4db6-ba82-dc11e89a7c00-kube-api-access-hvmhq\") pod \"e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn\" (UID: \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\") " pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" Nov 02 23:21:48.311255 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:48.311231 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1632e0b-a576-4db6-ba82-dc11e89a7c00-istio-token\") pod \"e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn\" (UID: \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\") " pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" Nov 02 23:21:48.372877 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:48.372841 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" Nov 02 23:21:48.796010 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:48.795977 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn"] Nov 02 23:21:48.855926 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:48.853225 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:21:48.855926 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:48.853324 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:21:48.855926 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:48.853367 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:21:49.494233 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:49.494196 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" event={"ID":"a1632e0b-a576-4db6-ba82-dc11e89a7c00","Type":"ContainerStarted","Data":"12e662bb18e88e5acd93940191e506762a5c8d60d33f2c366c5d70e535342593"} Nov 02 23:21:49.494693 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:49.494259 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" event={"ID":"a1632e0b-a576-4db6-ba82-dc11e89a7c00","Type":"ContainerStarted","Data":"198050dfbd6fd0d119dccff87bb6f702582c76253edfd7335b9beada5064b6e7"} Nov 02 23:21:49.660266 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:49.660191 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" podStartSLOduration=2.660169868 podStartE2EDuration="2.660169868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:21:49.653283963 +0000 UTC m=+1758.621815767" watchObservedRunningTime="2026-11-02 23:21:49.660169868 +0000 UTC m=+1758.628701657" Nov 02 23:21:50.373493 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:50.373460 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" Nov 02 23:21:50.378998 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:50.378966 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" Nov 02 23:21:50.406166 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:50.406136 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-5f96f7cdc9-5nrp5"] Nov 02 23:21:50.409380 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:50.409355 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-5f96f7cdc9-5nrp5" Nov 02 23:21:50.476430 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:50.476398 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-027689\"" Nov 02 23:21:50.476430 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:50.476398 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-027689-dockercfg-pfhhd\"" Nov 02 23:21:50.479657 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:50.479630 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-5f96f7cdc9-5nrp5"] Nov 02 23:21:50.500355 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:50.500331 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" Nov 02 23:21:50.501386 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:50.501365 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" Nov 02 23:21:50.516923 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:50.516866 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4xgc\" (UniqueName: \"kubernetes.io/projected/f12c3e68-ee99-4f7d-aeeb-445a9a13a5d3-kube-api-access-q4xgc\") pod \"payload-pre-processing-e2e-ipp-a-027689-5f96f7cdc9-5nrp5\" (UID: \"f12c3e68-ee99-4f7d-aeeb-445a9a13a5d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-5f96f7cdc9-5nrp5" Nov 02 23:21:50.517086 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:50.516934 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f12c3e68-ee99-4f7d-aeeb-445a9a13a5d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-027689-5f96f7cdc9-5nrp5\" (UID: \"f12c3e68-ee99-4f7d-aeeb-445a9a13a5d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-5f96f7cdc9-5nrp5" Nov 02 23:21:50.618097 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:50.618058 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4xgc\" (UniqueName: \"kubernetes.io/projected/f12c3e68-ee99-4f7d-aeeb-445a9a13a5d3-kube-api-access-q4xgc\") pod \"payload-pre-processing-e2e-ipp-a-027689-5f96f7cdc9-5nrp5\" (UID: \"f12c3e68-ee99-4f7d-aeeb-445a9a13a5d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-5f96f7cdc9-5nrp5" Nov 02 23:21:50.618284 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:50.618113 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f12c3e68-ee99-4f7d-aeeb-445a9a13a5d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-027689-5f96f7cdc9-5nrp5\" (UID: \"f12c3e68-ee99-4f7d-aeeb-445a9a13a5d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-5f96f7cdc9-5nrp5" Nov 02 23:21:50.618878 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:50.618858 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f12c3e68-ee99-4f7d-aeeb-445a9a13a5d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-027689-5f96f7cdc9-5nrp5\" (UID: \"f12c3e68-ee99-4f7d-aeeb-445a9a13a5d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-5f96f7cdc9-5nrp5" Nov 02 23:21:50.672109 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:50.672037 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4xgc\" (UniqueName: \"kubernetes.io/projected/f12c3e68-ee99-4f7d-aeeb-445a9a13a5d3-kube-api-access-q4xgc\") pod \"payload-pre-processing-e2e-ipp-a-027689-5f96f7cdc9-5nrp5\" (UID: \"f12c3e68-ee99-4f7d-aeeb-445a9a13a5d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-5f96f7cdc9-5nrp5" Nov 02 23:21:50.723058 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:50.723019 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-5f96f7cdc9-5nrp5" Nov 02 23:21:51.099188 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:51.099162 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-5f96f7cdc9-5nrp5"] Nov 02 23:21:51.099320 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:21:51.099307 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf12c3e68_ee99_4f7d_aeeb_445a9a13a5d3.slice/crio-7c8ded6ffea2afd7860cad9d5a8bd8a9ccc63a549c24a04d7e3372a47b320d67 WatchSource:0}: Error finding container 7c8ded6ffea2afd7860cad9d5a8bd8a9ccc63a549c24a04d7e3372a47b320d67: Status 404 returned error can't find the container with id 7c8ded6ffea2afd7860cad9d5a8bd8a9ccc63a549c24a04d7e3372a47b320d67 Nov 02 23:21:51.507819 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:51.507785 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-5f96f7cdc9-5nrp5" event={"ID":"f12c3e68-ee99-4f7d-aeeb-445a9a13a5d3","Type":"ContainerStarted","Data":"7c8ded6ffea2afd7860cad9d5a8bd8a9ccc63a549c24a04d7e3372a47b320d67"} Nov 02 23:21:54.533590 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:54.533551 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-5f96f7cdc9-5nrp5" event={"ID":"f12c3e68-ee99-4f7d-aeeb-445a9a13a5d3","Type":"ContainerStarted","Data":"3585040fc23a6ac9543abc73d032b0cd881c0f34080364cb2eed8fb09cf93277"} Nov 02 23:21:54.534035 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:54.533641 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-5f96f7cdc9-5nrp5" Nov 02 23:21:54.647963 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:54.647885 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-5f96f7cdc9-5nrp5" podStartSLOduration=2.286569943 podStartE2EDuration="4.647866033s" totalImagesPullingTime="2.36129609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:21:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:21:54.641722599 +0000 UTC m=+1763.610254383" watchObservedRunningTime="2026-11-02 23:21:54.647866033 +0000 UTC m=+1763.616397821" Nov 02 23:21:56.822359 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:56.822321 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-68e067-7b49597786-b7bwj"] Nov 02 23:21:56.822843 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:56.822499 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68e067-7b49597786-b7bwj" podUID="7cd608db-c160-4c05-b8d5-aa3cf80288b7" containerName="maas-api" containerID="cri-o://d24ab1384a2ef77a6454dd2924beecb51bcf7e1c5e76743d3c2e00197940b135" gracePeriod=30 Nov 02 23:21:57.161533 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:57.161501 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68e067-7b49597786-b7bwj" Nov 02 23:21:57.304146 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:57.304107 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cd608db-c160-4c05-b8d5-aa3cf80288b7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cd608db-c160-4c05-b8d5-aa3cf80288b7-metrics-certs\") pod \"7cd608db-c160-4c05-b8d5-aa3cf80288b7\" (UID: \"7cd608db-c160-4c05-b8d5-aa3cf80288b7\") " Nov 02 23:21:57.304310 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:57.304204 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cd608db-c160-4c05-b8d5-aa3cf80288b7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cd608db-c160-4c05-b8d5-aa3cf80288b7-openshift-service-ca\") pod \"7cd608db-c160-4c05-b8d5-aa3cf80288b7\" (UID: \"7cd608db-c160-4c05-b8d5-aa3cf80288b7\") " Nov 02 23:21:57.304396 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:57.304377 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cd608db-c160-4c05-b8d5-aa3cf80288b7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cd608db-c160-4c05-b8d5-aa3cf80288b7-maas-api-tls\") pod \"7cd608db-c160-4c05-b8d5-aa3cf80288b7\" (UID: \"7cd608db-c160-4c05-b8d5-aa3cf80288b7\") " Nov 02 23:21:57.304500 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:57.304481 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cd608db-c160-4c05-b8d5-aa3cf80288b7-kube-api-access-l8tkq\" (UniqueName: \"kubernetes.io/projected/7cd608db-c160-4c05-b8d5-aa3cf80288b7-kube-api-access-l8tkq\") pod \"7cd608db-c160-4c05-b8d5-aa3cf80288b7\" (UID: \"7cd608db-c160-4c05-b8d5-aa3cf80288b7\") " Nov 02 23:21:57.304623 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:57.304595 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cd608db-c160-4c05-b8d5-aa3cf80288b7-openshift-service-ca" pod "7cd608db-c160-4c05-b8d5-aa3cf80288b7" (UID: "7cd608db-c160-4c05-b8d5-aa3cf80288b7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:21:57.305006 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:57.304983 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cd608db-c160-4c05-b8d5-aa3cf80288b7-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:21:57.306419 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:57.306391 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cd608db-c160-4c05-b8d5-aa3cf80288b7-maas-api-tls" pod "7cd608db-c160-4c05-b8d5-aa3cf80288b7" (UID: "7cd608db-c160-4c05-b8d5-aa3cf80288b7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:21:57.306850 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:57.306823 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cd608db-c160-4c05-b8d5-aa3cf80288b7-metrics-certs" pod "7cd608db-c160-4c05-b8d5-aa3cf80288b7" (UID: "7cd608db-c160-4c05-b8d5-aa3cf80288b7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:21:57.306950 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:57.306854 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cd608db-c160-4c05-b8d5-aa3cf80288b7-kube-api-access-l8tkq" pod "7cd608db-c160-4c05-b8d5-aa3cf80288b7" (UID: "7cd608db-c160-4c05-b8d5-aa3cf80288b7"). InnerVolumeSpecName "kube-api-access-l8tkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:21:57.433984 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:57.431096 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cd608db-c160-4c05-b8d5-aa3cf80288b7-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:21:57.433984 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:57.431138 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8tkq\" (UniqueName: \"kubernetes.io/projected/7cd608db-c160-4c05-b8d5-aa3cf80288b7-kube-api-access-l8tkq\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:21:57.433984 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:57.431154 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cd608db-c160-4c05-b8d5-aa3cf80288b7-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:21:57.554672 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:57.554633 5076 generic.go:356] "Generic (PLEG): container finished" podID="7cd608db-c160-4c05-b8d5-aa3cf80288b7" containerID="d24ab1384a2ef77a6454dd2924beecb51bcf7e1c5e76743d3c2e00197940b135" exitCode=0 Nov 02 23:21:57.554861 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:57.554685 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68e067-7b49597786-b7bwj" Nov 02 23:21:57.554861 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:57.554716 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68e067-7b49597786-b7bwj" event={"ID":"7cd608db-c160-4c05-b8d5-aa3cf80288b7","Type":"ContainerDied","Data":"d24ab1384a2ef77a6454dd2924beecb51bcf7e1c5e76743d3c2e00197940b135"} Nov 02 23:21:57.554861 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:57.554781 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-68e067-7b49597786-b7bwj" event={"ID":"7cd608db-c160-4c05-b8d5-aa3cf80288b7","Type":"ContainerDied","Data":"3038436ced2ca0642daf2599a8700c43460d2dc84475306c845e8af8ff2a0e94"} Nov 02 23:21:57.554861 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:57.554814 5076 scope.go:122] "RemoveContainer" containerID="d24ab1384a2ef77a6454dd2924beecb51bcf7e1c5e76743d3c2e00197940b135" Nov 02 23:21:57.569116 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:57.569099 5076 scope.go:122] "RemoveContainer" containerID="d24ab1384a2ef77a6454dd2924beecb51bcf7e1c5e76743d3c2e00197940b135" Nov 02 23:21:57.569388 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:21:57.569371 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d24ab1384a2ef77a6454dd2924beecb51bcf7e1c5e76743d3c2e00197940b135\": container with ID starting with d24ab1384a2ef77a6454dd2924beecb51bcf7e1c5e76743d3c2e00197940b135 not found: ID does not exist" containerID="d24ab1384a2ef77a6454dd2924beecb51bcf7e1c5e76743d3c2e00197940b135" Nov 02 23:21:57.569428 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:57.569397 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d24ab1384a2ef77a6454dd2924beecb51bcf7e1c5e76743d3c2e00197940b135"} err="failed to get container status \"d24ab1384a2ef77a6454dd2924beecb51bcf7e1c5e76743d3c2e00197940b135\": rpc error: code = NotFound desc = could not find container \"d24ab1384a2ef77a6454dd2924beecb51bcf7e1c5e76743d3c2e00197940b135\": container with ID starting with d24ab1384a2ef77a6454dd2924beecb51bcf7e1c5e76743d3c2e00197940b135 not found: ID does not exist" Nov 02 23:21:57.711297 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:57.711134 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-68e067-7b49597786-b7bwj"] Nov 02 23:21:57.758279 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:57.758238 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-68e067-7b49597786-b7bwj"] Nov 02 23:21:59.149460 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:21:59.149411 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cd608db-c160-4c05-b8d5-aa3cf80288b7" path="/var/lib/kubelet/pods/7cd608db-c160-4c05-b8d5-aa3cf80288b7/volumes" Nov 02 23:22:05.541982 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:05.541948 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-5f96f7cdc9-5nrp5" Nov 02 23:22:07.658361 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:07.658327 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq"] Nov 02 23:22:07.658831 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:07.658587 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" podUID="e055ddbc-25c2-439e-85a6-b16c23688888" containerName="istio-proxy" containerID="cri-o://d1a23f72caa5a553b124a859897ef5ca4d0cc0e65d4e2ca96363c403e48eba3d" gracePeriod=30 Nov 02 23:22:09.883972 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:22:09.883933 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:22:10.568003 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:10.567973 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct"] Nov 02 23:22:10.568596 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:10.568581 5076 cpu_manager.go:414] "containerMap: removing container" podUID="7cd608db-c160-4c05-b8d5-aa3cf80288b7" containerName="maas-api" Nov 02 23:22:10.568666 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:10.568597 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cd608db-c160-4c05-b8d5-aa3cf80288b7" containerName="maas-api" Nov 02 23:22:10.568704 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:10.568700 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cd608db-c160-4c05-b8d5-aa3cf80288b7" containerName="maas-api" Nov 02 23:22:10.573357 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:10.573337 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" Nov 02 23:22:10.632950 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:10.629929 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-c0064f-openshift-default-dockercfg-9khvm\"" Nov 02 23:22:10.737909 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:10.723057 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7bcd988-c4a5-46e6-bba1-c4346f21922d-istio-token\") pod \"e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct\" (UID: \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\") " pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" Nov 02 23:22:10.749855 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:10.738253 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7bcd988-c4a5-46e6-bba1-c4346f21922d-istiod-ca-cert\") pod \"e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct\" (UID: \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\") " pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" Nov 02 23:22:10.765925 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:10.750254 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7bcd988-c4a5-46e6-bba1-c4346f21922d-istio-envoy\") pod \"e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct\" (UID: \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\") " pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" Nov 02 23:22:10.778917 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:10.766250 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sfr2\" (UniqueName: \"kubernetes.io/projected/c7bcd988-c4a5-46e6-bba1-c4346f21922d-kube-api-access-4sfr2\") pod \"e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct\" (UID: \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\") " pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" Nov 02 23:22:10.817909 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:10.786991 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7bcd988-c4a5-46e6-bba1-c4346f21922d-istio-data\") pod \"e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct\" (UID: \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\") " pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" Nov 02 23:22:10.818297 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:10.818223 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7bcd988-c4a5-46e6-bba1-c4346f21922d-credential-socket\") pod \"e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct\" (UID: \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\") " pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" Nov 02 23:22:10.818502 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:10.818482 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7bcd988-c4a5-46e6-bba1-c4346f21922d-workload-socket\") pod \"e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct\" (UID: \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\") " pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" Nov 02 23:22:10.831060 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:10.831022 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7bcd988-c4a5-46e6-bba1-c4346f21922d-workload-certs\") pod \"e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct\" (UID: \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\") " pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" Nov 02 23:22:10.832542 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:10.831318 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7bcd988-c4a5-46e6-bba1-c4346f21922d-istio-podinfo\") pod \"e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct\" (UID: \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\") " pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" Nov 02 23:22:10.832957 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:10.832553 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct"] Nov 02 23:22:10.934252 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:10.934217 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7bcd988-c4a5-46e6-bba1-c4346f21922d-istio-podinfo\") pod \"e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct\" (UID: \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\") " pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" Nov 02 23:22:10.934634 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:10.934273 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7bcd988-c4a5-46e6-bba1-c4346f21922d-istio-token\") pod \"e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct\" (UID: \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\") " pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" Nov 02 23:22:10.934634 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:10.934337 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7bcd988-c4a5-46e6-bba1-c4346f21922d-istiod-ca-cert\") pod \"e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct\" (UID: \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\") " pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" Nov 02 23:22:10.934634 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:10.934401 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7bcd988-c4a5-46e6-bba1-c4346f21922d-istio-envoy\") pod \"e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct\" (UID: \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\") " pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" Nov 02 23:22:10.934634 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:10.934449 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sfr2\" (UniqueName: \"kubernetes.io/projected/c7bcd988-c4a5-46e6-bba1-c4346f21922d-kube-api-access-4sfr2\") pod \"e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct\" (UID: \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\") " pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" Nov 02 23:22:10.934634 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:10.934468 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7bcd988-c4a5-46e6-bba1-c4346f21922d-istio-data\") pod \"e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct\" (UID: \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\") " pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" Nov 02 23:22:10.934634 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:10.934506 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7bcd988-c4a5-46e6-bba1-c4346f21922d-credential-socket\") pod \"e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct\" (UID: \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\") " pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" Nov 02 23:22:10.934634 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:10.934544 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7bcd988-c4a5-46e6-bba1-c4346f21922d-workload-socket\") pod \"e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct\" (UID: \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\") " pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" Nov 02 23:22:10.934634 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:10.934568 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7bcd988-c4a5-46e6-bba1-c4346f21922d-workload-certs\") pod \"e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct\" (UID: \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\") " pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" Nov 02 23:22:10.935044 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:10.935018 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7bcd988-c4a5-46e6-bba1-c4346f21922d-workload-certs\") pod \"e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct\" (UID: \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\") " pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" Nov 02 23:22:10.935163 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:10.935141 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7bcd988-c4a5-46e6-bba1-c4346f21922d-credential-socket\") pod \"e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct\" (UID: \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\") " pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" Nov 02 23:22:10.935291 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:10.935209 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7bcd988-c4a5-46e6-bba1-c4346f21922d-workload-socket\") pod \"e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct\" (UID: \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\") " pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" Nov 02 23:22:10.935291 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:10.935269 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7bcd988-c4a5-46e6-bba1-c4346f21922d-istio-data\") pod \"e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct\" (UID: \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\") " pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" Nov 02 23:22:10.935492 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:10.935472 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7bcd988-c4a5-46e6-bba1-c4346f21922d-istiod-ca-cert\") pod \"e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct\" (UID: \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\") " pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" Nov 02 23:22:10.937037 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:10.937018 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7bcd988-c4a5-46e6-bba1-c4346f21922d-istio-podinfo\") pod \"e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct\" (UID: \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\") " pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" Nov 02 23:22:10.937088 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:10.937049 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7bcd988-c4a5-46e6-bba1-c4346f21922d-istio-envoy\") pod \"e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct\" (UID: \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\") " pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" Nov 02 23:22:10.989074 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:10.989043 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7bcd988-c4a5-46e6-bba1-c4346f21922d-istio-token\") pod \"e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct\" (UID: \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\") " pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" Nov 02 23:22:10.989210 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:10.989094 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sfr2\" (UniqueName: \"kubernetes.io/projected/c7bcd988-c4a5-46e6-bba1-c4346f21922d-kube-api-access-4sfr2\") pod \"e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct\" (UID: \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\") " pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" Nov 02 23:22:11.186373 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:11.186341 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" Nov 02 23:22:11.613972 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:11.613938 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct"] Nov 02 23:22:11.614803 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:22:11.614770 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7bcd988_c4a5_46e6_bba1_c4346f21922d.slice/crio-af957e6f639869a01e32d0b8a3fbf78930b7837f662431c122a76adf09aa13f0 WatchSource:0}: Error finding container af957e6f639869a01e32d0b8a3fbf78930b7837f662431c122a76adf09aa13f0: Status 404 returned error can't find the container with id af957e6f639869a01e32d0b8a3fbf78930b7837f662431c122a76adf09aa13f0 Nov 02 23:22:11.617886 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:11.617853 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:22:11.618007 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:11.617963 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:22:11.618051 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:11.618014 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:22:11.687762 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:11.687733 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" event={"ID":"c7bcd988-c4a5-46e6-bba1-c4346f21922d","Type":"ContainerStarted","Data":"af957e6f639869a01e32d0b8a3fbf78930b7837f662431c122a76adf09aa13f0"} Nov 02 23:22:12.697205 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:12.697171 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" event={"ID":"c7bcd988-c4a5-46e6-bba1-c4346f21922d","Type":"ContainerStarted","Data":"3610146349edc6f65d4cd50dd26f467e09bb2d807ae0606d3e6dfd74516329f7"} Nov 02 23:22:12.862137 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:12.862060 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" podStartSLOduration=2.862040688 podStartE2EDuration="2.862040688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:22:12.858085795 +0000 UTC m=+1781.826617585" watchObservedRunningTime="2026-11-02 23:22:12.862040688 +0000 UTC m=+1781.830572472" Nov 02 23:22:12.956668 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:12.956642 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" Nov 02 23:22:13.061099 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.061063 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e055ddbc-25c2-439e-85a6-b16c23688888-istio-token\" (UniqueName: \"kubernetes.io/projected/e055ddbc-25c2-439e-85a6-b16c23688888-istio-token\") pod \"e055ddbc-25c2-439e-85a6-b16c23688888\" (UID: \"e055ddbc-25c2-439e-85a6-b16c23688888\") " Nov 02 23:22:13.061099 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.061100 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e055ddbc-25c2-439e-85a6-b16c23688888-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e055ddbc-25c2-439e-85a6-b16c23688888-istio-data\") pod \"e055ddbc-25c2-439e-85a6-b16c23688888\" (UID: \"e055ddbc-25c2-439e-85a6-b16c23688888\") " Nov 02 23:22:13.061344 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.061166 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e055ddbc-25c2-439e-85a6-b16c23688888-kube-api-access-q6cx8\" (UniqueName: \"kubernetes.io/projected/e055ddbc-25c2-439e-85a6-b16c23688888-kube-api-access-q6cx8\") pod \"e055ddbc-25c2-439e-85a6-b16c23688888\" (UID: \"e055ddbc-25c2-439e-85a6-b16c23688888\") " Nov 02 23:22:13.061344 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.061204 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e055ddbc-25c2-439e-85a6-b16c23688888-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e055ddbc-25c2-439e-85a6-b16c23688888-istio-podinfo\") pod \"e055ddbc-25c2-439e-85a6-b16c23688888\" (UID: \"e055ddbc-25c2-439e-85a6-b16c23688888\") " Nov 02 23:22:13.061344 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.061294 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e055ddbc-25c2-439e-85a6-b16c23688888-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e055ddbc-25c2-439e-85a6-b16c23688888-istio-envoy\") pod \"e055ddbc-25c2-439e-85a6-b16c23688888\" (UID: \"e055ddbc-25c2-439e-85a6-b16c23688888\") " Nov 02 23:22:13.061344 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.061339 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e055ddbc-25c2-439e-85a6-b16c23688888-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e055ddbc-25c2-439e-85a6-b16c23688888-workload-certs\") pod \"e055ddbc-25c2-439e-85a6-b16c23688888\" (UID: \"e055ddbc-25c2-439e-85a6-b16c23688888\") " Nov 02 23:22:13.061561 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.061389 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e055ddbc-25c2-439e-85a6-b16c23688888-istio-data" pod "e055ddbc-25c2-439e-85a6-b16c23688888" (UID: "e055ddbc-25c2-439e-85a6-b16c23688888"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:13.061561 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.061402 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e055ddbc-25c2-439e-85a6-b16c23688888-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e055ddbc-25c2-439e-85a6-b16c23688888-workload-socket\") pod \"e055ddbc-25c2-439e-85a6-b16c23688888\" (UID: \"e055ddbc-25c2-439e-85a6-b16c23688888\") " Nov 02 23:22:13.061561 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.061442 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e055ddbc-25c2-439e-85a6-b16c23688888-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e055ddbc-25c2-439e-85a6-b16c23688888-istiod-ca-cert\") pod \"e055ddbc-25c2-439e-85a6-b16c23688888\" (UID: \"e055ddbc-25c2-439e-85a6-b16c23688888\") " Nov 02 23:22:13.061561 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.061538 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e055ddbc-25c2-439e-85a6-b16c23688888-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e055ddbc-25c2-439e-85a6-b16c23688888-credential-socket\") pod \"e055ddbc-25c2-439e-85a6-b16c23688888\" (UID: \"e055ddbc-25c2-439e-85a6-b16c23688888\") " Nov 02 23:22:13.061764 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.061563 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e055ddbc-25c2-439e-85a6-b16c23688888-workload-certs" pod "e055ddbc-25c2-439e-85a6-b16c23688888" (UID: "e055ddbc-25c2-439e-85a6-b16c23688888"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:13.061764 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.061649 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e055ddbc-25c2-439e-85a6-b16c23688888-workload-socket" pod "e055ddbc-25c2-439e-85a6-b16c23688888" (UID: "e055ddbc-25c2-439e-85a6-b16c23688888"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:13.061867 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.061815 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e055ddbc-25c2-439e-85a6-b16c23688888-istiod-ca-cert" pod "e055ddbc-25c2-439e-85a6-b16c23688888" (UID: "e055ddbc-25c2-439e-85a6-b16c23688888"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:22:13.061968 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.061880 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e055ddbc-25c2-439e-85a6-b16c23688888-credential-socket" pod "e055ddbc-25c2-439e-85a6-b16c23688888" (UID: "e055ddbc-25c2-439e-85a6-b16c23688888"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:13.062255 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.062234 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e055ddbc-25c2-439e-85a6-b16c23688888-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:13.062331 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.062260 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e055ddbc-25c2-439e-85a6-b16c23688888-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:13.062331 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.062270 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e055ddbc-25c2-439e-85a6-b16c23688888-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:13.062331 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.062279 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e055ddbc-25c2-439e-85a6-b16c23688888-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:13.062331 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.062287 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e055ddbc-25c2-439e-85a6-b16c23688888-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:13.063577 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.063544 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e055ddbc-25c2-439e-85a6-b16c23688888-istio-envoy" pod "e055ddbc-25c2-439e-85a6-b16c23688888" (UID: "e055ddbc-25c2-439e-85a6-b16c23688888"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:13.063700 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.063686 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e055ddbc-25c2-439e-85a6-b16c23688888-kube-api-access-q6cx8" pod "e055ddbc-25c2-439e-85a6-b16c23688888" (UID: "e055ddbc-25c2-439e-85a6-b16c23688888"). InnerVolumeSpecName "kube-api-access-q6cx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:22:13.063881 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.063858 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e055ddbc-25c2-439e-85a6-b16c23688888-istio-podinfo" pod "e055ddbc-25c2-439e-85a6-b16c23688888" (UID: "e055ddbc-25c2-439e-85a6-b16c23688888"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:22:13.064034 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.063874 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e055ddbc-25c2-439e-85a6-b16c23688888-istio-token" pod "e055ddbc-25c2-439e-85a6-b16c23688888" (UID: "e055ddbc-25c2-439e-85a6-b16c23688888"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:22:13.163082 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.163055 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e055ddbc-25c2-439e-85a6-b16c23688888-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:13.163082 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.163083 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e055ddbc-25c2-439e-85a6-b16c23688888-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:13.163230 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.163093 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e055ddbc-25c2-439e-85a6-b16c23688888-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:13.163230 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.163102 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6cx8\" (UniqueName: \"kubernetes.io/projected/e055ddbc-25c2-439e-85a6-b16c23688888-kube-api-access-q6cx8\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:13.187122 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.187086 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" Nov 02 23:22:13.193148 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.193122 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" Nov 02 23:22:13.708076 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.708040 5076 generic.go:356] "Generic (PLEG): container finished" podID="e055ddbc-25c2-439e-85a6-b16c23688888" containerID="d1a23f72caa5a553b124a859897ef5ca4d0cc0e65d4e2ca96363c403e48eba3d" exitCode=0 Nov 02 23:22:13.708509 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.708102 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" Nov 02 23:22:13.708509 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.708123 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" event={"ID":"e055ddbc-25c2-439e-85a6-b16c23688888","Type":"ContainerDied","Data":"d1a23f72caa5a553b124a859897ef5ca4d0cc0e65d4e2ca96363c403e48eba3d"} Nov 02 23:22:13.708509 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.708186 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq" event={"ID":"e055ddbc-25c2-439e-85a6-b16c23688888","Type":"ContainerDied","Data":"8692f8cdbbabcfbff6154c42adaa0d64e14f30d2dfbe1f892981508ab03c3883"} Nov 02 23:22:13.708509 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.708220 5076 scope.go:122] "RemoveContainer" containerID="d1a23f72caa5a553b124a859897ef5ca4d0cc0e65d4e2ca96363c403e48eba3d" Nov 02 23:22:13.708754 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.708653 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" Nov 02 23:22:13.710315 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.710297 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" Nov 02 23:22:13.720053 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.720025 5076 scope.go:122] "RemoveContainer" containerID="bb00f3e50511a389808878d36acc49d0ce70589eb29effdc2dee1cf0ce556f2e" Nov 02 23:22:13.731534 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.731510 5076 scope.go:122] "RemoveContainer" containerID="d1a23f72caa5a553b124a859897ef5ca4d0cc0e65d4e2ca96363c403e48eba3d" Nov 02 23:22:13.731866 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:22:13.731843 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1a23f72caa5a553b124a859897ef5ca4d0cc0e65d4e2ca96363c403e48eba3d\": container with ID starting with d1a23f72caa5a553b124a859897ef5ca4d0cc0e65d4e2ca96363c403e48eba3d not found: ID does not exist" containerID="d1a23f72caa5a553b124a859897ef5ca4d0cc0e65d4e2ca96363c403e48eba3d" Nov 02 23:22:13.731933 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.731876 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1a23f72caa5a553b124a859897ef5ca4d0cc0e65d4e2ca96363c403e48eba3d"} err="failed to get container status \"d1a23f72caa5a553b124a859897ef5ca4d0cc0e65d4e2ca96363c403e48eba3d\": rpc error: code = NotFound desc = could not find container \"d1a23f72caa5a553b124a859897ef5ca4d0cc0e65d4e2ca96363c403e48eba3d\": container with ID starting with d1a23f72caa5a553b124a859897ef5ca4d0cc0e65d4e2ca96363c403e48eba3d not found: ID does not exist" Nov 02 23:22:13.731933 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.731911 5076 scope.go:122] "RemoveContainer" containerID="bb00f3e50511a389808878d36acc49d0ce70589eb29effdc2dee1cf0ce556f2e" Nov 02 23:22:13.732172 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:22:13.732155 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb00f3e50511a389808878d36acc49d0ce70589eb29effdc2dee1cf0ce556f2e\": container with ID starting with bb00f3e50511a389808878d36acc49d0ce70589eb29effdc2dee1cf0ce556f2e not found: ID does not exist" containerID="bb00f3e50511a389808878d36acc49d0ce70589eb29effdc2dee1cf0ce556f2e" Nov 02 23:22:13.732211 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.732176 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb00f3e50511a389808878d36acc49d0ce70589eb29effdc2dee1cf0ce556f2e"} err="failed to get container status \"bb00f3e50511a389808878d36acc49d0ce70589eb29effdc2dee1cf0ce556f2e\": rpc error: code = NotFound desc = could not find container \"bb00f3e50511a389808878d36acc49d0ce70589eb29effdc2dee1cf0ce556f2e\": container with ID starting with bb00f3e50511a389808878d36acc49d0ce70589eb29effdc2dee1cf0ce556f2e not found: ID does not exist" Nov 02 23:22:13.869139 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.869099 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq"] Nov 02 23:22:13.915494 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:13.915454 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-68e067-openshift-default-5dff57fd77-dqltq"] Nov 02 23:22:14.921468 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:14.921422 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-818862-6f5ffd459f-q9ddl"] Nov 02 23:22:15.023206 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:15.023172 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-818862-9f98cbf95-vwbzx"] Nov 02 23:22:15.152182 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:15.152150 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e055ddbc-25c2-439e-85a6-b16c23688888" path="/var/lib/kubelet/pods/e055ddbc-25c2-439e-85a6-b16c23688888/volumes" Nov 02 23:22:19.957919 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:19.957848 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-818862-6f5ffd459f-q9ddl" podUID="6d833e3b-de7a-42b4-8d36-0d8f1b7659ca" containerName="payload-processing" containerID="cri-o://6e5b48632097918c4c2a0c77e51a1cdf9f66a3c1a96abefde0c285eab43f575d" gracePeriod=25 Nov 02 23:22:20.024171 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:20.023984 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-818862-9f98cbf95-vwbzx" podUID="49aee3c8-3339-4424-a3da-b81c4731e66a" containerName="payload-pre-processing" containerID="cri-o://83371c0399feb68d40f17c5e66d967ad6ab961bdbc2f2608249bcb442db72766" gracePeriod=25 Nov 02 23:22:20.439305 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:20.439271 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-818862-6f5ffd459f-q9ddl" Nov 02 23:22:20.471830 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:20.471773 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-818862-9f98cbf95-vwbzx" Nov 02 23:22:20.530781 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:20.530753 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49aee3c8-3339-4424-a3da-b81c4731e66a-kube-api-access-9gd7s\" (UniqueName: \"kubernetes.io/projected/49aee3c8-3339-4424-a3da-b81c4731e66a-kube-api-access-9gd7s\") pod \"49aee3c8-3339-4424-a3da-b81c4731e66a\" (UID: \"49aee3c8-3339-4424-a3da-b81c4731e66a\") " Nov 02 23:22:20.530970 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:20.530804 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49aee3c8-3339-4424-a3da-b81c4731e66a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49aee3c8-3339-4424-a3da-b81c4731e66a-plugins-config-volume\") pod \"49aee3c8-3339-4424-a3da-b81c4731e66a\" (UID: \"49aee3c8-3339-4424-a3da-b81c4731e66a\") " Nov 02 23:22:20.530970 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:20.530852 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d833e3b-de7a-42b4-8d36-0d8f1b7659ca-kube-api-access-dm7dd\" (UniqueName: \"kubernetes.io/projected/6d833e3b-de7a-42b4-8d36-0d8f1b7659ca-kube-api-access-dm7dd\") pod \"6d833e3b-de7a-42b4-8d36-0d8f1b7659ca\" (UID: \"6d833e3b-de7a-42b4-8d36-0d8f1b7659ca\") " Nov 02 23:22:20.531091 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:20.531081 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d833e3b-de7a-42b4-8d36-0d8f1b7659ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d833e3b-de7a-42b4-8d36-0d8f1b7659ca-plugins-config-volume\") pod \"6d833e3b-de7a-42b4-8d36-0d8f1b7659ca\" (UID: \"6d833e3b-de7a-42b4-8d36-0d8f1b7659ca\") " Nov 02 23:22:20.531248 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:20.531219 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49aee3c8-3339-4424-a3da-b81c4731e66a-plugins-config-volume" pod "49aee3c8-3339-4424-a3da-b81c4731e66a" (UID: "49aee3c8-3339-4424-a3da-b81c4731e66a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:22:20.531515 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:20.531485 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d833e3b-de7a-42b4-8d36-0d8f1b7659ca-plugins-config-volume" pod "6d833e3b-de7a-42b4-8d36-0d8f1b7659ca" (UID: "6d833e3b-de7a-42b4-8d36-0d8f1b7659ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:22:20.531515 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:20.531503 5076 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49aee3c8-3339-4424-a3da-b81c4731e66a-plugins-config-volume\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:20.533034 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:20.533015 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49aee3c8-3339-4424-a3da-b81c4731e66a-kube-api-access-9gd7s" pod "49aee3c8-3339-4424-a3da-b81c4731e66a" (UID: "49aee3c8-3339-4424-a3da-b81c4731e66a"). InnerVolumeSpecName "kube-api-access-9gd7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:22:20.533100 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:20.533055 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d833e3b-de7a-42b4-8d36-0d8f1b7659ca-kube-api-access-dm7dd" pod "6d833e3b-de7a-42b4-8d36-0d8f1b7659ca" (UID: "6d833e3b-de7a-42b4-8d36-0d8f1b7659ca"). InnerVolumeSpecName "kube-api-access-dm7dd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:22:20.632564 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:20.632517 5076 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d833e3b-de7a-42b4-8d36-0d8f1b7659ca-plugins-config-volume\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:20.632564 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:20.632561 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gd7s\" (UniqueName: \"kubernetes.io/projected/49aee3c8-3339-4424-a3da-b81c4731e66a-kube-api-access-9gd7s\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:20.632564 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:20.632574 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm7dd\" (UniqueName: \"kubernetes.io/projected/6d833e3b-de7a-42b4-8d36-0d8f1b7659ca-kube-api-access-dm7dd\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:20.776528 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:20.776439 5076 generic.go:356] "Generic (PLEG): container finished" podID="49aee3c8-3339-4424-a3da-b81c4731e66a" containerID="83371c0399feb68d40f17c5e66d967ad6ab961bdbc2f2608249bcb442db72766" exitCode=0 Nov 02 23:22:20.776528 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:20.776483 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-818862-9f98cbf95-vwbzx" Nov 02 23:22:20.776528 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:20.776513 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-818862-9f98cbf95-vwbzx" event={"ID":"49aee3c8-3339-4424-a3da-b81c4731e66a","Type":"ContainerDied","Data":"83371c0399feb68d40f17c5e66d967ad6ab961bdbc2f2608249bcb442db72766"} Nov 02 23:22:20.776802 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:20.776567 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-818862-9f98cbf95-vwbzx" event={"ID":"49aee3c8-3339-4424-a3da-b81c4731e66a","Type":"ContainerDied","Data":"d4a6c114b57cc298e9c7b2322f5fd546a59a90a974d2a67adc9c0461283131cf"} Nov 02 23:22:20.776802 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:20.776595 5076 scope.go:122] "RemoveContainer" containerID="83371c0399feb68d40f17c5e66d967ad6ab961bdbc2f2608249bcb442db72766" Nov 02 23:22:20.778172 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:20.778148 5076 generic.go:356] "Generic (PLEG): container finished" podID="6d833e3b-de7a-42b4-8d36-0d8f1b7659ca" containerID="6e5b48632097918c4c2a0c77e51a1cdf9f66a3c1a96abefde0c285eab43f575d" exitCode=0 Nov 02 23:22:20.778287 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:20.778184 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-818862-6f5ffd459f-q9ddl" Nov 02 23:22:20.778287 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:20.778235 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-818862-6f5ffd459f-q9ddl" event={"ID":"6d833e3b-de7a-42b4-8d36-0d8f1b7659ca","Type":"ContainerDied","Data":"6e5b48632097918c4c2a0c77e51a1cdf9f66a3c1a96abefde0c285eab43f575d"} Nov 02 23:22:20.778287 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:20.778274 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-818862-6f5ffd459f-q9ddl" event={"ID":"6d833e3b-de7a-42b4-8d36-0d8f1b7659ca","Type":"ContainerDied","Data":"f5058a849c3ba8dd57142926c33e7e4a4d0ee38707028c64f9b03a5a9608a422"} Nov 02 23:22:20.787476 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:20.787434 5076 scope.go:122] "RemoveContainer" containerID="83371c0399feb68d40f17c5e66d967ad6ab961bdbc2f2608249bcb442db72766" Nov 02 23:22:20.787757 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:22:20.787730 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83371c0399feb68d40f17c5e66d967ad6ab961bdbc2f2608249bcb442db72766\": container with ID starting with 83371c0399feb68d40f17c5e66d967ad6ab961bdbc2f2608249bcb442db72766 not found: ID does not exist" containerID="83371c0399feb68d40f17c5e66d967ad6ab961bdbc2f2608249bcb442db72766" Nov 02 23:22:20.787838 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:20.787768 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83371c0399feb68d40f17c5e66d967ad6ab961bdbc2f2608249bcb442db72766"} err="failed to get container status \"83371c0399feb68d40f17c5e66d967ad6ab961bdbc2f2608249bcb442db72766\": rpc error: code = NotFound desc = could not find container \"83371c0399feb68d40f17c5e66d967ad6ab961bdbc2f2608249bcb442db72766\": container with ID starting with 83371c0399feb68d40f17c5e66d967ad6ab961bdbc2f2608249bcb442db72766 not found: ID does not exist" Nov 02 23:22:20.787838 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:20.787787 5076 scope.go:122] "RemoveContainer" containerID="6e5b48632097918c4c2a0c77e51a1cdf9f66a3c1a96abefde0c285eab43f575d" Nov 02 23:22:20.800994 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:20.800974 5076 scope.go:122] "RemoveContainer" containerID="6e5b48632097918c4c2a0c77e51a1cdf9f66a3c1a96abefde0c285eab43f575d" Nov 02 23:22:20.801328 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:22:20.801305 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e5b48632097918c4c2a0c77e51a1cdf9f66a3c1a96abefde0c285eab43f575d\": container with ID starting with 6e5b48632097918c4c2a0c77e51a1cdf9f66a3c1a96abefde0c285eab43f575d not found: ID does not exist" containerID="6e5b48632097918c4c2a0c77e51a1cdf9f66a3c1a96abefde0c285eab43f575d" Nov 02 23:22:20.801400 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:20.801339 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e5b48632097918c4c2a0c77e51a1cdf9f66a3c1a96abefde0c285eab43f575d"} err="failed to get container status \"6e5b48632097918c4c2a0c77e51a1cdf9f66a3c1a96abefde0c285eab43f575d\": rpc error: code = NotFound desc = could not find container \"6e5b48632097918c4c2a0c77e51a1cdf9f66a3c1a96abefde0c285eab43f575d\": container with ID starting with 6e5b48632097918c4c2a0c77e51a1cdf9f66a3c1a96abefde0c285eab43f575d not found: ID does not exist" Nov 02 23:22:20.933626 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:20.933592 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-818862-6f5ffd459f-q9ddl"] Nov 02 23:22:20.937343 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:20.937319 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-818862-6f5ffd459f-q9ddl"] Nov 02 23:22:21.041743 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:21.041654 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-818862-9f98cbf95-vwbzx"] Nov 02 23:22:21.044737 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:21.044709 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-818862-9f98cbf95-vwbzx"] Nov 02 23:22:21.149019 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:21.148984 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49aee3c8-3339-4424-a3da-b81c4731e66a" path="/var/lib/kubelet/pods/49aee3c8-3339-4424-a3da-b81c4731e66a/volumes" Nov 02 23:22:21.149547 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:21.149525 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d833e3b-de7a-42b4-8d36-0d8f1b7659ca" path="/var/lib/kubelet/pods/6d833e3b-de7a-42b4-8d36-0d8f1b7659ca/volumes" Nov 02 23:22:30.966755 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:30.966719 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-77cd547f9f-tmkt6"] Nov 02 23:22:30.967414 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:30.967396 5076 cpu_manager.go:414] "containerMap: removing container" podUID="6d833e3b-de7a-42b4-8d36-0d8f1b7659ca" containerName="payload-processing" Nov 02 23:22:30.967511 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:30.967418 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d833e3b-de7a-42b4-8d36-0d8f1b7659ca" containerName="payload-processing" Nov 02 23:22:30.967511 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:30.967433 5076 cpu_manager.go:414] "containerMap: removing container" podUID="e055ddbc-25c2-439e-85a6-b16c23688888" containerName="istio-proxy" Nov 02 23:22:30.967511 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:30.967438 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e055ddbc-25c2-439e-85a6-b16c23688888" containerName="istio-proxy" Nov 02 23:22:30.967511 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:30.967504 5076 cpu_manager.go:414] "containerMap: removing container" podUID="49aee3c8-3339-4424-a3da-b81c4731e66a" containerName="payload-pre-processing" Nov 02 23:22:30.967511 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:30.967510 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49aee3c8-3339-4424-a3da-b81c4731e66a" containerName="payload-pre-processing" Nov 02 23:22:30.967752 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:30.967522 5076 cpu_manager.go:414] "containerMap: removing container" podUID="e055ddbc-25c2-439e-85a6-b16c23688888" containerName="istio-proxy" Nov 02 23:22:30.967752 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:30.967527 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e055ddbc-25c2-439e-85a6-b16c23688888" containerName="istio-proxy" Nov 02 23:22:30.967752 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:30.967645 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d833e3b-de7a-42b4-8d36-0d8f1b7659ca" containerName="payload-processing" Nov 02 23:22:30.967752 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:30.967657 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="e055ddbc-25c2-439e-85a6-b16c23688888" containerName="istio-proxy" Nov 02 23:22:30.967752 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:30.967678 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="49aee3c8-3339-4424-a3da-b81c4731e66a" containerName="payload-pre-processing" Nov 02 23:22:30.971481 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:30.971463 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-77cd547f9f-tmkt6" Nov 02 23:22:31.024076 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:31.024044 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-77cd547f9f-tmkt6"] Nov 02 23:22:31.067621 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:31.067570 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a69f65\"" Nov 02 23:22:31.067802 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:31.067698 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a69f65-dockercfg-k5764\"" Nov 02 23:22:31.108961 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:31.108921 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l87gh\" (UniqueName: \"kubernetes.io/projected/3df93a02-4d85-4282-891a-fbb18a0e7e33-kube-api-access-l87gh\") pod \"payload-pre-processing-e2e-ipp-a-a69f65-77cd547f9f-tmkt6\" (UID: \"3df93a02-4d85-4282-891a-fbb18a0e7e33\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-77cd547f9f-tmkt6" Nov 02 23:22:31.109122 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:31.108997 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3df93a02-4d85-4282-891a-fbb18a0e7e33-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a69f65-77cd547f9f-tmkt6\" (UID: \"3df93a02-4d85-4282-891a-fbb18a0e7e33\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-77cd547f9f-tmkt6" Nov 02 23:22:31.210732 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:31.210697 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l87gh\" (UniqueName: \"kubernetes.io/projected/3df93a02-4d85-4282-891a-fbb18a0e7e33-kube-api-access-l87gh\") pod \"payload-pre-processing-e2e-ipp-a-a69f65-77cd547f9f-tmkt6\" (UID: \"3df93a02-4d85-4282-891a-fbb18a0e7e33\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-77cd547f9f-tmkt6" Nov 02 23:22:31.210925 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:31.210760 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3df93a02-4d85-4282-891a-fbb18a0e7e33-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a69f65-77cd547f9f-tmkt6\" (UID: \"3df93a02-4d85-4282-891a-fbb18a0e7e33\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-77cd547f9f-tmkt6" Nov 02 23:22:31.259350 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:31.259275 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a69f65\"" Nov 02 23:22:31.261920 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:31.261878 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3df93a02-4d85-4282-891a-fbb18a0e7e33-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a69f65-77cd547f9f-tmkt6\" (UID: \"3df93a02-4d85-4282-891a-fbb18a0e7e33\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-77cd547f9f-tmkt6" Nov 02 23:22:31.264852 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:31.264825 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l87gh\" (UniqueName: \"kubernetes.io/projected/3df93a02-4d85-4282-891a-fbb18a0e7e33-kube-api-access-l87gh\") pod \"payload-pre-processing-e2e-ipp-a-a69f65-77cd547f9f-tmkt6\" (UID: \"3df93a02-4d85-4282-891a-fbb18a0e7e33\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-77cd547f9f-tmkt6" Nov 02 23:22:31.334369 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:31.334335 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a69f65-dockercfg-k5764\"" Nov 02 23:22:31.336428 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:31.336408 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-77cd547f9f-tmkt6" Nov 02 23:22:31.694743 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:31.694721 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 02 23:22:31.710380 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:31.710349 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-77cd547f9f-tmkt6"] Nov 02 23:22:31.725111 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:31.725088 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a69f65-6f9d977d6d-jcxjn"] Nov 02 23:22:31.725965 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:31.725943 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="e055ddbc-25c2-439e-85a6-b16c23688888" containerName="istio-proxy" Nov 02 23:22:31.729780 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:31.729763 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a69f65-6f9d977d6d-jcxjn" Nov 02 23:22:31.744091 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:31.744068 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 02 23:22:31.808882 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:31.808853 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a69f65-6f9d977d6d-jcxjn"] Nov 02 23:22:31.820726 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:31.820696 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428ec111-bd83-465d-b67e-e3ffcfdc946c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a69f65-6f9d977d6d-jcxjn\" (UID: \"428ec111-bd83-465d-b67e-e3ffcfdc946c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a69f65-6f9d977d6d-jcxjn" Nov 02 23:22:31.820906 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:31.820773 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnxdn\" (UniqueName: \"kubernetes.io/projected/428ec111-bd83-465d-b67e-e3ffcfdc946c-kube-api-access-qnxdn\") pod \"payload-processing-e2e-ipp-a-a69f65-6f9d977d6d-jcxjn\" (UID: \"428ec111-bd83-465d-b67e-e3ffcfdc946c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a69f65-6f9d977d6d-jcxjn" Nov 02 23:22:31.858970 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:31.858930 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-77cd547f9f-tmkt6" event={"ID":"3df93a02-4d85-4282-891a-fbb18a0e7e33","Type":"ContainerStarted","Data":"b2bfcf4f756bc5cc5e8c58e91714c641d283e148b6c38914c9fae5eac16969ad"} Nov 02 23:22:31.859148 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:31.858997 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-77cd547f9f-tmkt6" event={"ID":"3df93a02-4d85-4282-891a-fbb18a0e7e33","Type":"ContainerStarted","Data":"bba83565f94e5d43cc5e6cf85257a32e233c6750535ca8fa8fd556a97a176415"} Nov 02 23:22:31.859148 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:31.859027 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-77cd547f9f-tmkt6" Nov 02 23:22:31.921937 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:31.921843 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428ec111-bd83-465d-b67e-e3ffcfdc946c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a69f65-6f9d977d6d-jcxjn\" (UID: \"428ec111-bd83-465d-b67e-e3ffcfdc946c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a69f65-6f9d977d6d-jcxjn" Nov 02 23:22:31.922084 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:31.921957 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnxdn\" (UniqueName: \"kubernetes.io/projected/428ec111-bd83-465d-b67e-e3ffcfdc946c-kube-api-access-qnxdn\") pod \"payload-processing-e2e-ipp-a-a69f65-6f9d977d6d-jcxjn\" (UID: \"428ec111-bd83-465d-b67e-e3ffcfdc946c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a69f65-6f9d977d6d-jcxjn" Nov 02 23:22:31.922557 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:31.922538 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428ec111-bd83-465d-b67e-e3ffcfdc946c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a69f65-6f9d977d6d-jcxjn\" (UID: \"428ec111-bd83-465d-b67e-e3ffcfdc946c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a69f65-6f9d977d6d-jcxjn" Nov 02 23:22:31.967870 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:31.967812 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-77cd547f9f-tmkt6" podStartSLOduration=1.967797416 podStartE2EDuration="1.967797416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:22:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:22:31.966833354 +0000 UTC m=+1800.935365139" watchObservedRunningTime="2026-11-02 23:22:31.967797416 +0000 UTC m=+1800.936329261" Nov 02 23:22:31.979384 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:31.979354 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnxdn\" (UniqueName: \"kubernetes.io/projected/428ec111-bd83-465d-b67e-e3ffcfdc946c-kube-api-access-qnxdn\") pod \"payload-processing-e2e-ipp-a-a69f65-6f9d977d6d-jcxjn\" (UID: \"428ec111-bd83-465d-b67e-e3ffcfdc946c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a69f65-6f9d977d6d-jcxjn" Nov 02 23:22:32.076510 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:32.076469 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a69f65-6f9d977d6d-jcxjn" Nov 02 23:22:32.536168 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:32.536138 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a69f65-6f9d977d6d-jcxjn"] Nov 02 23:22:32.537370 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:22:32.537336 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod428ec111_bd83_465d_b67e_e3ffcfdc946c.slice/crio-ea9da1527d40aa74ed07e7c1cbcfaaa1ba4aca415fa246b8d511aff29e94f21a WatchSource:0}: Error finding container ea9da1527d40aa74ed07e7c1cbcfaaa1ba4aca415fa246b8d511aff29e94f21a: Status 404 returned error can't find the container with id ea9da1527d40aa74ed07e7c1cbcfaaa1ba4aca415fa246b8d511aff29e94f21a Nov 02 23:22:32.867210 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:32.867111 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a69f65-6f9d977d6d-jcxjn" event={"ID":"428ec111-bd83-465d-b67e-e3ffcfdc946c","Type":"ContainerStarted","Data":"b49d83af05e72ba178d9a1e9d4657fe08241b179e9c29df695b83be503998ae4"} Nov 02 23:22:32.867210 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:32.867166 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a69f65-6f9d977d6d-jcxjn" event={"ID":"428ec111-bd83-465d-b67e-e3ffcfdc946c","Type":"ContainerStarted","Data":"ea9da1527d40aa74ed07e7c1cbcfaaa1ba4aca415fa246b8d511aff29e94f21a"} Nov 02 23:22:32.867400 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:32.867218 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a69f65-6f9d977d6d-jcxjn" Nov 02 23:22:32.977063 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:32.977013 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a69f65-6f9d977d6d-jcxjn" podStartSLOduration=1.976996906 podStartE2EDuration="1.976996906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:22:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:22:32.976503607 +0000 UTC m=+1801.945035391" watchObservedRunningTime="2026-11-02 23:22:32.976996906 +0000 UTC m=+1801.945528690" Nov 02 23:22:36.732591 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:36.732552 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8"] Nov 02 23:22:36.733052 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:36.732754 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8" podUID="fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7" containerName="maas-api" containerID="cri-o://8338582e008b6c98f9f04b58530758c1666cc13a57f7ee57ff512271f87346ee" gracePeriod=30 Nov 02 23:22:36.899548 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:36.899510 5076 generic.go:356] "Generic (PLEG): container finished" podID="fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7" containerID="8338582e008b6c98f9f04b58530758c1666cc13a57f7ee57ff512271f87346ee" exitCode=0 Nov 02 23:22:36.899723 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:36.899585 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8" event={"ID":"fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7","Type":"ContainerDied","Data":"8338582e008b6c98f9f04b58530758c1666cc13a57f7ee57ff512271f87346ee"} Nov 02 23:22:37.079260 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:37.079236 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8" Nov 02 23:22:37.252911 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:37.251690 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7-maas-api-tls\") pod \"fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7\" (UID: \"fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7\") " Nov 02 23:22:37.252911 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:37.252666 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7-metrics-certs\") pod \"fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7\" (UID: \"fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7\") " Nov 02 23:22:37.252911 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:37.252823 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7-openshift-service-ca\") pod \"fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7\" (UID: \"fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7\") " Nov 02 23:22:37.253631 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:37.253584 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7-openshift-service-ca" pod "fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7" (UID: "fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:22:37.254684 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:37.254152 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7-kube-api-access-fb6z9\" (UniqueName: \"kubernetes.io/projected/fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7-kube-api-access-fb6z9\") pod \"fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7\" (UID: \"fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7\") " Nov 02 23:22:37.256866 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:37.256708 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:37.259613 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:37.259554 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7-metrics-certs" pod "fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7" (UID: "fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:22:37.263656 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:37.263633 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7-kube-api-access-fb6z9" pod "fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7" (UID: "fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7"). InnerVolumeSpecName "kube-api-access-fb6z9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:22:37.263775 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:37.263633 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7-maas-api-tls" pod "fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7" (UID: "fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:22:37.363007 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:37.361047 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:37.363007 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:37.361080 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:37.363007 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:37.361094 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb6z9\" (UniqueName: \"kubernetes.io/projected/fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7-kube-api-access-fb6z9\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:37.906935 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:37.906902 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8" Nov 02 23:22:37.907403 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:37.906943 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8" event={"ID":"fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7","Type":"ContainerDied","Data":"8f4d55421f317ad74fbf7e63cce73b1d2fd91c3fc4c1cf39f4a2778505cfc81f"} Nov 02 23:22:37.907403 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:37.907021 5076 scope.go:122] "RemoveContainer" containerID="8338582e008b6c98f9f04b58530758c1666cc13a57f7ee57ff512271f87346ee" Nov 02 23:22:38.065505 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:38.065458 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8"] Nov 02 23:22:38.112312 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:38.112249 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e953257b-649d4c7ccc-kwpf8"] Nov 02 23:22:39.139166 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:39.139128 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js"] Nov 02 23:22:39.139700 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:39.139337 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" podUID="21678735-6a1a-4654-8b9d-b979a78520ff" containerName="istio-proxy" containerID="cri-o://9be3896d0fac297b0bb0bf2099051b9a07bc34bba7c3a101a473f1754dee03c9" gracePeriod=30 Nov 02 23:22:39.149679 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:39.149642 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7" path="/var/lib/kubelet/pods/fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7/volumes" Nov 02 23:22:39.157568 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:39.157518 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs"] Nov 02 23:22:39.157992 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:39.157962 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" podUID="984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc" containerName="istio-proxy" containerID="cri-o://6fad4738f09acc85561dcb3395f302dbc05dfba5ce94019d530b0acbb2c15da8" gracePeriod=30 Nov 02 23:22:39.178636 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:39.178602 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr"] Nov 02 23:22:39.178821 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:39.178800 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" podUID="6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd" containerName="istio-proxy" containerID="cri-o://bca75fd4bc4583c7bbd11161cd219d69b53399e71a8f0b8d628b68ccc56e46e0" gracePeriod=30 Nov 02 23:22:39.884539 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:22:39.884496 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:22:40.497958 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:40.497922 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm"] Nov 02 23:22:40.498681 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:40.498654 5076 cpu_manager.go:414] "containerMap: removing container" podUID="fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7" containerName="maas-api" Nov 02 23:22:40.498681 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:40.498678 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7" containerName="maas-api" Nov 02 23:22:40.498841 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:40.498786 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc28255d-a0e6-493a-bf25-5c0e6b8ba6c7" containerName="maas-api" Nov 02 23:22:40.504031 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:40.504009 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" Nov 02 23:22:40.608972 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:40.595760 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm"] Nov 02 23:22:40.626265 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:40.626234 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-b57963-openshift-default-dockercfg-x8vr9\"" Nov 02 23:22:40.671597 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:40.659931 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd27984-010f-441c-a3f4-553933c0529a-workload-socket\") pod \"e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm\" (UID: \"dfd27984-010f-441c-a3f4-553933c0529a\") " pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" Nov 02 23:22:40.672198 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:40.672130 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfd27984-010f-441c-a3f4-553933c0529a-istiod-ca-cert\") pod \"e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm\" (UID: \"dfd27984-010f-441c-a3f4-553933c0529a\") " pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" Nov 02 23:22:40.672549 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:40.672503 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfd27984-010f-441c-a3f4-553933c0529a-istio-token\") pod \"e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm\" (UID: \"dfd27984-010f-441c-a3f4-553933c0529a\") " pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" Nov 02 23:22:40.672695 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:40.672676 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2jbn\" (UniqueName: \"kubernetes.io/projected/dfd27984-010f-441c-a3f4-553933c0529a-kube-api-access-f2jbn\") pod \"e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm\" (UID: \"dfd27984-010f-441c-a3f4-553933c0529a\") " pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" Nov 02 23:22:40.672981 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:40.672962 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfd27984-010f-441c-a3f4-553933c0529a-istio-envoy\") pod \"e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm\" (UID: \"dfd27984-010f-441c-a3f4-553933c0529a\") " pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" Nov 02 23:22:40.673131 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:40.673116 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfd27984-010f-441c-a3f4-553933c0529a-istio-podinfo\") pod \"e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm\" (UID: \"dfd27984-010f-441c-a3f4-553933c0529a\") " pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" Nov 02 23:22:40.673407 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:40.673383 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfd27984-010f-441c-a3f4-553933c0529a-istio-data\") pod \"e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm\" (UID: \"dfd27984-010f-441c-a3f4-553933c0529a\") " pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" Nov 02 23:22:40.681462 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:40.673673 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfd27984-010f-441c-a3f4-553933c0529a-workload-certs\") pod \"e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm\" (UID: \"dfd27984-010f-441c-a3f4-553933c0529a\") " pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" Nov 02 23:22:40.684064 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:40.684027 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd27984-010f-441c-a3f4-553933c0529a-credential-socket\") pod \"e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm\" (UID: \"dfd27984-010f-441c-a3f4-553933c0529a\") " pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" Nov 02 23:22:40.798263 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:40.796526 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfd27984-010f-441c-a3f4-553933c0529a-istio-data\") pod \"e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm\" (UID: \"dfd27984-010f-441c-a3f4-553933c0529a\") " pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" Nov 02 23:22:40.798263 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:40.796626 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfd27984-010f-441c-a3f4-553933c0529a-workload-certs\") pod \"e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm\" (UID: \"dfd27984-010f-441c-a3f4-553933c0529a\") " pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" Nov 02 23:22:40.798263 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:40.796667 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd27984-010f-441c-a3f4-553933c0529a-credential-socket\") pod \"e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm\" (UID: \"dfd27984-010f-441c-a3f4-553933c0529a\") " pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" Nov 02 23:22:40.798263 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:40.796704 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd27984-010f-441c-a3f4-553933c0529a-workload-socket\") pod \"e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm\" (UID: \"dfd27984-010f-441c-a3f4-553933c0529a\") " pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" Nov 02 23:22:40.798263 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:40.796720 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfd27984-010f-441c-a3f4-553933c0529a-istiod-ca-cert\") pod \"e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm\" (UID: \"dfd27984-010f-441c-a3f4-553933c0529a\") " pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" Nov 02 23:22:40.798263 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:40.796750 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfd27984-010f-441c-a3f4-553933c0529a-istio-token\") pod \"e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm\" (UID: \"dfd27984-010f-441c-a3f4-553933c0529a\") " pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" Nov 02 23:22:40.798263 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:40.796767 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2jbn\" (UniqueName: \"kubernetes.io/projected/dfd27984-010f-441c-a3f4-553933c0529a-kube-api-access-f2jbn\") pod \"e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm\" (UID: \"dfd27984-010f-441c-a3f4-553933c0529a\") " pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" Nov 02 23:22:40.798263 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:40.796809 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfd27984-010f-441c-a3f4-553933c0529a-istio-envoy\") pod \"e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm\" (UID: \"dfd27984-010f-441c-a3f4-553933c0529a\") " pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" Nov 02 23:22:40.798263 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:40.796854 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfd27984-010f-441c-a3f4-553933c0529a-istio-podinfo\") pod \"e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm\" (UID: \"dfd27984-010f-441c-a3f4-553933c0529a\") " pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" Nov 02 23:22:40.798263 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:40.797599 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd27984-010f-441c-a3f4-553933c0529a-workload-socket\") pod \"e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm\" (UID: \"dfd27984-010f-441c-a3f4-553933c0529a\") " pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" Nov 02 23:22:40.798263 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:40.797798 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfd27984-010f-441c-a3f4-553933c0529a-istio-data\") pod \"e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm\" (UID: \"dfd27984-010f-441c-a3f4-553933c0529a\") " pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" Nov 02 23:22:40.798263 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:40.798064 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfd27984-010f-441c-a3f4-553933c0529a-workload-certs\") pod \"e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm\" (UID: \"dfd27984-010f-441c-a3f4-553933c0529a\") " pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" Nov 02 23:22:40.798997 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:40.798569 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd27984-010f-441c-a3f4-553933c0529a-credential-socket\") pod \"e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm\" (UID: \"dfd27984-010f-441c-a3f4-553933c0529a\") " pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" Nov 02 23:22:40.799051 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:40.799029 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfd27984-010f-441c-a3f4-553933c0529a-istiod-ca-cert\") pod \"e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm\" (UID: \"dfd27984-010f-441c-a3f4-553933c0529a\") " pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" Nov 02 23:22:40.802097 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:40.802039 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfd27984-010f-441c-a3f4-553933c0529a-istio-envoy\") pod \"e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm\" (UID: \"dfd27984-010f-441c-a3f4-553933c0529a\") " pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" Nov 02 23:22:40.807270 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:40.807247 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfd27984-010f-441c-a3f4-553933c0529a-istio-podinfo\") pod \"e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm\" (UID: \"dfd27984-010f-441c-a3f4-553933c0529a\") " pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" Nov 02 23:22:40.853010 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:40.852972 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfd27984-010f-441c-a3f4-553933c0529a-istio-token\") pod \"e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm\" (UID: \"dfd27984-010f-441c-a3f4-553933c0529a\") " pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" Nov 02 23:22:40.853400 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:40.853375 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2jbn\" (UniqueName: \"kubernetes.io/projected/dfd27984-010f-441c-a3f4-553933c0529a-kube-api-access-f2jbn\") pod \"e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm\" (UID: \"dfd27984-010f-441c-a3f4-553933c0529a\") " pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" Nov 02 23:22:41.149516 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:41.149432 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" Nov 02 23:22:41.588925 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:41.588881 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm"] Nov 02 23:22:41.589318 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:22:41.589149 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd27984_010f_441c_a3f4_553933c0529a.slice/crio-15de44a64194381386ca89dc001c5be00985cc781a476fc7b44384c9030d06d0 WatchSource:0}: Error finding container 15de44a64194381386ca89dc001c5be00985cc781a476fc7b44384c9030d06d0: Status 404 returned error can't find the container with id 15de44a64194381386ca89dc001c5be00985cc781a476fc7b44384c9030d06d0 Nov 02 23:22:41.591748 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:41.591717 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:22:41.591829 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:41.591784 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:22:41.591829 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:41.591813 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:22:41.940114 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:41.940070 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" event={"ID":"dfd27984-010f-441c-a3f4-553933c0529a","Type":"ContainerStarted","Data":"70c2c4ebc1f046f648bf9e4350e65bc41dfb010749d084db3dc8cff0ef6e30c4"} Nov 02 23:22:41.940346 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:41.940150 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" event={"ID":"dfd27984-010f-441c-a3f4-553933c0529a","Type":"ContainerStarted","Data":"15de44a64194381386ca89dc001c5be00985cc781a476fc7b44384c9030d06d0"} Nov 02 23:22:42.053704 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:42.053652 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" podStartSLOduration=2.053636106 podStartE2EDuration="2.053636106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:22:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:22:42.051460299 +0000 UTC m=+1811.019992084" watchObservedRunningTime="2026-11-02 23:22:42.053636106 +0000 UTC m=+1811.022167891" Nov 02 23:22:42.150092 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:42.150056 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" Nov 02 23:22:42.155499 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:42.155475 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" Nov 02 23:22:42.869584 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:42.869552 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-77cd547f9f-tmkt6" Nov 02 23:22:42.950708 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:42.950677 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" Nov 02 23:22:42.952086 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:42.952060 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" Nov 02 23:22:43.876549 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:43.876517 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a69f65-6f9d977d6d-jcxjn" Nov 02 23:22:44.462831 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.462799 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" Nov 02 23:22:44.523808 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.523785 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" Nov 02 23:22:44.536912 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.536876 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" Nov 02 23:22:44.565771 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.565745 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21678735-6a1a-4654-8b9d-b979a78520ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21678735-6a1a-4654-8b9d-b979a78520ff-istio-podinfo\") pod \"21678735-6a1a-4654-8b9d-b979a78520ff\" (UID: \"21678735-6a1a-4654-8b9d-b979a78520ff\") " Nov 02 23:22:44.565945 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.565786 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21678735-6a1a-4654-8b9d-b979a78520ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21678735-6a1a-4654-8b9d-b979a78520ff-workload-socket\") pod \"21678735-6a1a-4654-8b9d-b979a78520ff\" (UID: \"21678735-6a1a-4654-8b9d-b979a78520ff\") " Nov 02 23:22:44.565945 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.565836 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-workload-certs\") pod \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\" (UID: \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\") " Nov 02 23:22:44.565945 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.565884 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21678735-6a1a-4654-8b9d-b979a78520ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21678735-6a1a-4654-8b9d-b979a78520ff-istiod-ca-cert\") pod \"21678735-6a1a-4654-8b9d-b979a78520ff\" (UID: \"21678735-6a1a-4654-8b9d-b979a78520ff\") " Nov 02 23:22:44.565945 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.565938 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21678735-6a1a-4654-8b9d-b979a78520ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21678735-6a1a-4654-8b9d-b979a78520ff-workload-certs\") pod \"21678735-6a1a-4654-8b9d-b979a78520ff\" (UID: \"21678735-6a1a-4654-8b9d-b979a78520ff\") " Nov 02 23:22:44.566170 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.565955 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-istio-token\" (UniqueName: \"kubernetes.io/projected/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-istio-token\") pod \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\" (UID: \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\") " Nov 02 23:22:44.566170 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.565995 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-istio-envoy\") pod \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\" (UID: \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\") " Nov 02 23:22:44.566170 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.566078 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-istio-data\") pod \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\" (UID: \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\") " Nov 02 23:22:44.566170 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.566125 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-istio-podinfo\") pod \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\" (UID: \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\") " Nov 02 23:22:44.566170 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.566130 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21678735-6a1a-4654-8b9d-b979a78520ff-workload-socket" pod "21678735-6a1a-4654-8b9d-b979a78520ff" (UID: "21678735-6a1a-4654-8b9d-b979a78520ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:44.566170 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.566118 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-workload-certs" pod "984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc" (UID: "984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:44.566170 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.566153 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-kube-api-access-f8gpb\" (UniqueName: \"kubernetes.io/projected/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-kube-api-access-f8gpb\") pod \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\" (UID: \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\") " Nov 02 23:22:44.566511 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.566198 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-workload-socket\") pod \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\" (UID: \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\") " Nov 02 23:22:44.566511 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.566231 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21678735-6a1a-4654-8b9d-b979a78520ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21678735-6a1a-4654-8b9d-b979a78520ff-credential-socket\") pod \"21678735-6a1a-4654-8b9d-b979a78520ff\" (UID: \"21678735-6a1a-4654-8b9d-b979a78520ff\") " Nov 02 23:22:44.566511 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.566258 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21678735-6a1a-4654-8b9d-b979a78520ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21678735-6a1a-4654-8b9d-b979a78520ff-istio-envoy\") pod \"21678735-6a1a-4654-8b9d-b979a78520ff\" (UID: \"21678735-6a1a-4654-8b9d-b979a78520ff\") " Nov 02 23:22:44.566511 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.566311 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21678735-6a1a-4654-8b9d-b979a78520ff-istiod-ca-cert" pod "21678735-6a1a-4654-8b9d-b979a78520ff" (UID: "21678735-6a1a-4654-8b9d-b979a78520ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:22:44.566511 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.566357 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21678735-6a1a-4654-8b9d-b979a78520ff-kube-api-access-hp5zc\" (UniqueName: \"kubernetes.io/projected/21678735-6a1a-4654-8b9d-b979a78520ff-kube-api-access-hp5zc\") pod \"21678735-6a1a-4654-8b9d-b979a78520ff\" (UID: \"21678735-6a1a-4654-8b9d-b979a78520ff\") " Nov 02 23:22:44.566511 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.566383 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-istio-data" pod "984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc" (UID: "984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:44.566511 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.566448 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-credential-socket\") pod \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\" (UID: \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\") " Nov 02 23:22:44.566511 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.566511 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21678735-6a1a-4654-8b9d-b979a78520ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21678735-6a1a-4654-8b9d-b979a78520ff-istio-data\") pod \"21678735-6a1a-4654-8b9d-b979a78520ff\" (UID: \"21678735-6a1a-4654-8b9d-b979a78520ff\") " Nov 02 23:22:44.566997 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.566524 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21678735-6a1a-4654-8b9d-b979a78520ff-workload-certs" pod "21678735-6a1a-4654-8b9d-b979a78520ff" (UID: "21678735-6a1a-4654-8b9d-b979a78520ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:44.566997 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.566554 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-workload-socket" pod "984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc" (UID: "984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:44.566997 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.566603 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21678735-6a1a-4654-8b9d-b979a78520ff-credential-socket" pod "21678735-6a1a-4654-8b9d-b979a78520ff" (UID: "21678735-6a1a-4654-8b9d-b979a78520ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:44.566997 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.566633 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-istiod-ca-cert\") pod \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\" (UID: \"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc\") " Nov 02 23:22:44.566997 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.566676 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21678735-6a1a-4654-8b9d-b979a78520ff-istio-token\" (UniqueName: \"kubernetes.io/projected/21678735-6a1a-4654-8b9d-b979a78520ff-istio-token\") pod \"21678735-6a1a-4654-8b9d-b979a78520ff\" (UID: \"21678735-6a1a-4654-8b9d-b979a78520ff\") " Nov 02 23:22:44.566997 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.566742 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21678735-6a1a-4654-8b9d-b979a78520ff-istio-data" pod "21678735-6a1a-4654-8b9d-b979a78520ff" (UID: "21678735-6a1a-4654-8b9d-b979a78520ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:44.566997 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.566948 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-credential-socket" pod "984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc" (UID: "984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:44.567370 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.567346 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21678735-6a1a-4654-8b9d-b979a78520ff-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.567437 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.567369 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.567437 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.567385 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21678735-6a1a-4654-8b9d-b979a78520ff-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.567437 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.567399 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21678735-6a1a-4654-8b9d-b979a78520ff-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.567437 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.567413 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.567437 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.567427 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.567723 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.567440 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21678735-6a1a-4654-8b9d-b979a78520ff-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.567723 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.567457 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.567723 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.567470 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21678735-6a1a-4654-8b9d-b979a78520ff-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.567955 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.567815 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-istiod-ca-cert" pod "984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc" (UID: "984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:22:44.568671 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.568650 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-istio-token" pod "984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc" (UID: "984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:22:44.568780 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.568672 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21678735-6a1a-4654-8b9d-b979a78520ff-istio-podinfo" pod "21678735-6a1a-4654-8b9d-b979a78520ff" (UID: "21678735-6a1a-4654-8b9d-b979a78520ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:22:44.568780 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.568738 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-istio-podinfo" pod "984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc" (UID: "984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:22:44.569071 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.569041 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-kube-api-access-f8gpb" pod "984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc" (UID: "984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc"). InnerVolumeSpecName "kube-api-access-f8gpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:22:44.569169 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.569148 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21678735-6a1a-4654-8b9d-b979a78520ff-kube-api-access-hp5zc" pod "21678735-6a1a-4654-8b9d-b979a78520ff" (UID: "21678735-6a1a-4654-8b9d-b979a78520ff"). InnerVolumeSpecName "kube-api-access-hp5zc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:22:44.569218 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.569170 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21678735-6a1a-4654-8b9d-b979a78520ff-istio-envoy" pod "21678735-6a1a-4654-8b9d-b979a78520ff" (UID: "21678735-6a1a-4654-8b9d-b979a78520ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:44.569928 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.569885 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21678735-6a1a-4654-8b9d-b979a78520ff-istio-token" pod "21678735-6a1a-4654-8b9d-b979a78520ff" (UID: "21678735-6a1a-4654-8b9d-b979a78520ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:22:44.570178 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.570161 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-istio-envoy" pod "984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc" (UID: "984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:44.668493 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.668461 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-istio-data\") pod \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\" (UID: \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\") " Nov 02 23:22:44.668689 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.668528 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-credential-socket\") pod \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\" (UID: \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\") " Nov 02 23:22:44.668689 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.668595 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-workload-certs\") pod \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\" (UID: \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\") " Nov 02 23:22:44.668818 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.668735 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-istiod-ca-cert\") pod \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\" (UID: \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\") " Nov 02 23:22:44.668818 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.668775 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-istio-token\" (UniqueName: \"kubernetes.io/projected/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-istio-token\") pod \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\" (UID: \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\") " Nov 02 23:22:44.668818 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.668776 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-istio-data" pod "6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd" (UID: "6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:44.669016 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.668842 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-istio-envoy\") pod \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\" (UID: \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\") " Nov 02 23:22:44.669016 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.668940 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-workload-socket\") pod \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\" (UID: \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\") " Nov 02 23:22:44.669128 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.669016 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-istio-podinfo\") pod \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\" (UID: \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\") " Nov 02 23:22:44.669128 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.669054 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-kube-api-access-bt522\" (UniqueName: \"kubernetes.io/projected/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-kube-api-access-bt522\") pod \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\" (UID: \"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd\") " Nov 02 23:22:44.669236 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.669129 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-istiod-ca-cert" pod "6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd" (UID: "6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:22:44.669385 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.669363 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-workload-socket" pod "6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd" (UID: "6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:44.669817 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.669628 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21678735-6a1a-4654-8b9d-b979a78520ff-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.669817 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.669652 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.669817 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.669668 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp5zc\" (UniqueName: \"kubernetes.io/projected/21678735-6a1a-4654-8b9d-b979a78520ff-kube-api-access-hp5zc\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.669817 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.669685 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.669817 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.669700 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.669817 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.669713 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21678735-6a1a-4654-8b9d-b979a78520ff-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.669817 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.669726 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21678735-6a1a-4654-8b9d-b979a78520ff-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.669817 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.669739 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.669817 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.669752 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.669817 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.669766 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.669817 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.669762 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-workload-certs" pod "6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd" (UID: "6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:44.669817 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.669780 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.669817 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.669793 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8gpb\" (UniqueName: \"kubernetes.io/projected/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc-kube-api-access-f8gpb\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.669817 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.669819 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-credential-socket" pod "6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd" (UID: "6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:44.671955 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.671926 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-istio-podinfo" pod "6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd" (UID: "6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:22:44.671955 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.671939 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-istio-envoy" pod "6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd" (UID: "6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:44.672076 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.671978 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-kube-api-access-bt522" pod "6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd" (UID: "6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd"). InnerVolumeSpecName "kube-api-access-bt522". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:22:44.673416 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.673394 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-istio-token" pod "6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd" (UID: "6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:22:44.770832 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.770797 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.770832 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.770829 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.770832 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.770839 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.771092 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.770848 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.771092 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.770857 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.771092 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.770865 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt522\" (UniqueName: \"kubernetes.io/projected/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd-kube-api-access-bt522\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.963631 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.963594 5076 generic.go:356] "Generic (PLEG): container finished" podID="6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd" containerID="bca75fd4bc4583c7bbd11161cd219d69b53399e71a8f0b8d628b68ccc56e46e0" exitCode=0 Nov 02 23:22:44.964114 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.963644 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" Nov 02 23:22:44.964114 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.963680 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" event={"ID":"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd","Type":"ContainerDied","Data":"bca75fd4bc4583c7bbd11161cd219d69b53399e71a8f0b8d628b68ccc56e46e0"} Nov 02 23:22:44.964114 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.963754 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr" event={"ID":"6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd","Type":"ContainerDied","Data":"d39420625dd5d56b7d9616cca1a1bc80865950dd74ed4eea7fad9752ef4916ff"} Nov 02 23:22:44.964114 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.963803 5076 scope.go:122] "RemoveContainer" containerID="bca75fd4bc4583c7bbd11161cd219d69b53399e71a8f0b8d628b68ccc56e46e0" Nov 02 23:22:44.965244 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.965222 5076 generic.go:356] "Generic (PLEG): container finished" podID="984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc" containerID="6fad4738f09acc85561dcb3395f302dbc05dfba5ce94019d530b0acbb2c15da8" exitCode=0 Nov 02 23:22:44.965371 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.965264 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" Nov 02 23:22:44.965371 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.965302 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" event={"ID":"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc","Type":"ContainerDied","Data":"6fad4738f09acc85561dcb3395f302dbc05dfba5ce94019d530b0acbb2c15da8"} Nov 02 23:22:44.965371 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.965348 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs" event={"ID":"984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc","Type":"ContainerDied","Data":"a3b6be039a284bddf057cba46ffc58fedb8e9446c432bce72cea40e2e6f6fa24"} Nov 02 23:22:44.966877 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.966854 5076 generic.go:356] "Generic (PLEG): container finished" podID="21678735-6a1a-4654-8b9d-b979a78520ff" containerID="9be3896d0fac297b0bb0bf2099051b9a07bc34bba7c3a101a473f1754dee03c9" exitCode=0 Nov 02 23:22:44.967024 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.966915 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" Nov 02 23:22:44.967024 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.966917 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" event={"ID":"21678735-6a1a-4654-8b9d-b979a78520ff","Type":"ContainerDied","Data":"9be3896d0fac297b0bb0bf2099051b9a07bc34bba7c3a101a473f1754dee03c9"} Nov 02 23:22:44.967024 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.966966 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js" event={"ID":"21678735-6a1a-4654-8b9d-b979a78520ff","Type":"ContainerDied","Data":"a4e86138fa90fb7a44887610c22e5b557ca11c62e831fb00e8abef37ec9630be"} Nov 02 23:22:44.975524 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.975322 5076 scope.go:122] "RemoveContainer" containerID="bca75fd4bc4583c7bbd11161cd219d69b53399e71a8f0b8d628b68ccc56e46e0" Nov 02 23:22:44.975808 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:22:44.975764 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bca75fd4bc4583c7bbd11161cd219d69b53399e71a8f0b8d628b68ccc56e46e0\": container with ID starting with bca75fd4bc4583c7bbd11161cd219d69b53399e71a8f0b8d628b68ccc56e46e0 not found: ID does not exist" containerID="bca75fd4bc4583c7bbd11161cd219d69b53399e71a8f0b8d628b68ccc56e46e0" Nov 02 23:22:44.975869 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.975820 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bca75fd4bc4583c7bbd11161cd219d69b53399e71a8f0b8d628b68ccc56e46e0"} err="failed to get container status \"bca75fd4bc4583c7bbd11161cd219d69b53399e71a8f0b8d628b68ccc56e46e0\": rpc error: code = NotFound desc = could not find container \"bca75fd4bc4583c7bbd11161cd219d69b53399e71a8f0b8d628b68ccc56e46e0\": container with ID starting with bca75fd4bc4583c7bbd11161cd219d69b53399e71a8f0b8d628b68ccc56e46e0 not found: ID does not exist" Nov 02 23:22:44.975869 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.975846 5076 scope.go:122] "RemoveContainer" containerID="6fad4738f09acc85561dcb3395f302dbc05dfba5ce94019d530b0acbb2c15da8" Nov 02 23:22:44.985886 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.985865 5076 scope.go:122] "RemoveContainer" containerID="6fad4738f09acc85561dcb3395f302dbc05dfba5ce94019d530b0acbb2c15da8" Nov 02 23:22:44.986165 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:22:44.986148 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fad4738f09acc85561dcb3395f302dbc05dfba5ce94019d530b0acbb2c15da8\": container with ID starting with 6fad4738f09acc85561dcb3395f302dbc05dfba5ce94019d530b0acbb2c15da8 not found: ID does not exist" containerID="6fad4738f09acc85561dcb3395f302dbc05dfba5ce94019d530b0acbb2c15da8" Nov 02 23:22:44.986255 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.986175 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fad4738f09acc85561dcb3395f302dbc05dfba5ce94019d530b0acbb2c15da8"} err="failed to get container status \"6fad4738f09acc85561dcb3395f302dbc05dfba5ce94019d530b0acbb2c15da8\": rpc error: code = NotFound desc = could not find container \"6fad4738f09acc85561dcb3395f302dbc05dfba5ce94019d530b0acbb2c15da8\": container with ID starting with 6fad4738f09acc85561dcb3395f302dbc05dfba5ce94019d530b0acbb2c15da8 not found: ID does not exist" Nov 02 23:22:44.986255 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.986198 5076 scope.go:122] "RemoveContainer" containerID="9be3896d0fac297b0bb0bf2099051b9a07bc34bba7c3a101a473f1754dee03c9" Nov 02 23:22:44.996480 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.996251 5076 scope.go:122] "RemoveContainer" containerID="9be3896d0fac297b0bb0bf2099051b9a07bc34bba7c3a101a473f1754dee03c9" Nov 02 23:22:44.996803 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:22:44.996784 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9be3896d0fac297b0bb0bf2099051b9a07bc34bba7c3a101a473f1754dee03c9\": container with ID starting with 9be3896d0fac297b0bb0bf2099051b9a07bc34bba7c3a101a473f1754dee03c9 not found: ID does not exist" containerID="9be3896d0fac297b0bb0bf2099051b9a07bc34bba7c3a101a473f1754dee03c9" Nov 02 23:22:44.996869 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:44.996808 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9be3896d0fac297b0bb0bf2099051b9a07bc34bba7c3a101a473f1754dee03c9"} err="failed to get container status \"9be3896d0fac297b0bb0bf2099051b9a07bc34bba7c3a101a473f1754dee03c9\": rpc error: code = NotFound desc = could not find container \"9be3896d0fac297b0bb0bf2099051b9a07bc34bba7c3a101a473f1754dee03c9\": container with ID starting with 9be3896d0fac297b0bb0bf2099051b9a07bc34bba7c3a101a473f1754dee03c9 not found: ID does not exist" Nov 02 23:22:45.124198 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:45.124114 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs"] Nov 02 23:22:45.130222 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:45.130191 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-2p4gs"] Nov 02 23:22:45.149206 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:45.149177 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc" path="/var/lib/kubelet/pods/984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc/volumes" Nov 02 23:22:45.235938 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:45.235906 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr"] Nov 02 23:22:45.242735 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:45.242708 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-brzwr"] Nov 02 23:22:45.348738 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:45.348707 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js"] Nov 02 23:22:45.353323 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:45.353296 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-mp4js"] Nov 02 23:22:45.953163 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:45.953120 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr"] Nov 02 23:22:45.953814 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:45.953793 5076 cpu_manager.go:414] "containerMap: removing container" podUID="21678735-6a1a-4654-8b9d-b979a78520ff" containerName="istio-proxy" Nov 02 23:22:45.953939 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:45.953814 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21678735-6a1a-4654-8b9d-b979a78520ff" containerName="istio-proxy" Nov 02 23:22:45.953939 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:45.953837 5076 cpu_manager.go:414] "containerMap: removing container" podUID="984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc" containerName="istio-proxy" Nov 02 23:22:45.953939 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:45.953845 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc" containerName="istio-proxy" Nov 02 23:22:45.953939 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:45.953866 5076 cpu_manager.go:414] "containerMap: removing container" podUID="6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd" containerName="istio-proxy" Nov 02 23:22:45.953939 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:45.953874 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd" containerName="istio-proxy" Nov 02 23:22:45.954193 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:45.954031 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="984e5b3f-c720-44ac-9dfd-12ce2b4dc2bc" containerName="istio-proxy" Nov 02 23:22:45.954193 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:45.954050 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd" containerName="istio-proxy" Nov 02 23:22:45.954193 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:45.954072 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="21678735-6a1a-4654-8b9d-b979a78520ff" containerName="istio-proxy" Nov 02 23:22:45.959241 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:45.959220 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr" Nov 02 23:22:46.008985 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:46.008956 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-6a894c-serving-cert\"" Nov 02 23:22:46.013673 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:46.013642 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr"] Nov 02 23:22:46.087982 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:46.087946 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh2k6\" (UniqueName: \"kubernetes.io/projected/6ec4a99a-59d0-422c-98f4-066c5e3eb1a1-kube-api-access-kh2k6\") pod \"maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr\" (UID: \"6ec4a99a-59d0-422c-98f4-066c5e3eb1a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr" Nov 02 23:22:46.088375 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:46.088334 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ec4a99a-59d0-422c-98f4-066c5e3eb1a1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr\" (UID: \"6ec4a99a-59d0-422c-98f4-066c5e3eb1a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr" Nov 02 23:22:46.088487 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:46.088443 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ec4a99a-59d0-422c-98f4-066c5e3eb1a1-metrics-certs\") pod \"maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr\" (UID: \"6ec4a99a-59d0-422c-98f4-066c5e3eb1a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr" Nov 02 23:22:46.088487 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:46.088477 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ec4a99a-59d0-422c-98f4-066c5e3eb1a1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr\" (UID: \"6ec4a99a-59d0-422c-98f4-066c5e3eb1a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr" Nov 02 23:22:46.189354 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:46.189317 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ec4a99a-59d0-422c-98f4-066c5e3eb1a1-metrics-certs\") pod \"maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr\" (UID: \"6ec4a99a-59d0-422c-98f4-066c5e3eb1a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr" Nov 02 23:22:46.189354 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:46.189360 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ec4a99a-59d0-422c-98f4-066c5e3eb1a1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr\" (UID: \"6ec4a99a-59d0-422c-98f4-066c5e3eb1a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr" Nov 02 23:22:46.189612 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:46.189438 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh2k6\" (UniqueName: \"kubernetes.io/projected/6ec4a99a-59d0-422c-98f4-066c5e3eb1a1-kube-api-access-kh2k6\") pod \"maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr\" (UID: \"6ec4a99a-59d0-422c-98f4-066c5e3eb1a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr" Nov 02 23:22:46.189612 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:46.189576 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ec4a99a-59d0-422c-98f4-066c5e3eb1a1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr\" (UID: \"6ec4a99a-59d0-422c-98f4-066c5e3eb1a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr" Nov 02 23:22:46.189839 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:22:46.189813 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-6a894c-serving-cert: secret "maas-api-e2e-ipp-a-6a894c-serving-cert" not found Nov 02 23:22:46.190039 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:22:46.190017 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ec4a99a-59d0-422c-98f4-066c5e3eb1a1-maas-api-tls podName:6ec4a99a-59d0-422c-98f4-066c5e3eb1a1 nodeName:}" failed. No retries permitted until 2026-11-02 23:22:46.689995753 +0000 UTC m=+1815.658527535 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6ec4a99a-59d0-422c-98f4-066c5e3eb1a1-maas-api-tls") pod "maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr" (UID: "6ec4a99a-59d0-422c-98f4-066c5e3eb1a1") : secret "maas-api-e2e-ipp-a-6a894c-serving-cert" not found Nov 02 23:22:46.190159 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:46.190130 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ec4a99a-59d0-422c-98f4-066c5e3eb1a1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr\" (UID: \"6ec4a99a-59d0-422c-98f4-066c5e3eb1a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr" Nov 02 23:22:46.226955 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:46.226580 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ec4a99a-59d0-422c-98f4-066c5e3eb1a1-metrics-certs\") pod \"maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr\" (UID: \"6ec4a99a-59d0-422c-98f4-066c5e3eb1a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr" Nov 02 23:22:46.254595 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:46.254561 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh2k6\" (UniqueName: \"kubernetes.io/projected/6ec4a99a-59d0-422c-98f4-066c5e3eb1a1-kube-api-access-kh2k6\") pod \"maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr\" (UID: \"6ec4a99a-59d0-422c-98f4-066c5e3eb1a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr" Nov 02 23:22:46.698693 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:46.698658 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ec4a99a-59d0-422c-98f4-066c5e3eb1a1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr\" (UID: \"6ec4a99a-59d0-422c-98f4-066c5e3eb1a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr" Nov 02 23:22:46.701267 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:46.701232 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ec4a99a-59d0-422c-98f4-066c5e3eb1a1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr\" (UID: \"6ec4a99a-59d0-422c-98f4-066c5e3eb1a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr" Nov 02 23:22:46.872885 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:46.872855 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr" Nov 02 23:22:47.148899 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:47.148817 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21678735-6a1a-4654-8b9d-b979a78520ff" path="/var/lib/kubelet/pods/21678735-6a1a-4654-8b9d-b979a78520ff/volumes" Nov 02 23:22:47.149328 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:47.149313 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd" path="/var/lib/kubelet/pods/6c011db9-8ca8-46bd-a9d5-e33b8ce1dfdd/volumes" Nov 02 23:22:47.243575 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:47.243533 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr"] Nov 02 23:22:47.991347 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:47.991313 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr" event={"ID":"6ec4a99a-59d0-422c-98f4-066c5e3eb1a1","Type":"ContainerStarted","Data":"4a3001cc99cda7571beb9a7ffca53fb9f118845e2864340df33074013544f53b"} Nov 02 23:22:49.000315 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:49.000240 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr" event={"ID":"6ec4a99a-59d0-422c-98f4-066c5e3eb1a1","Type":"ContainerStarted","Data":"058d3ac6c51a8c272e3c09e095b0525a1df6e4e8e09ea97ac1ef61f429020737"} Nov 02 23:22:49.000787 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:49.000365 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr" Nov 02 23:22:49.110783 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:49.110733 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr" podStartSLOduration=3.037454152 podStartE2EDuration="4.110720135s" totalImagesPullingTime="1.073265983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:22:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:22:49.109372775 +0000 UTC m=+1818.077904559" watchObservedRunningTime="2026-11-02 23:22:49.110720135 +0000 UTC m=+1818.079251918" Nov 02 23:22:55.013546 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:22:55.013508 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr" Nov 02 23:23:06.947461 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:06.947423 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-818862-c954974c4-2vb2t"] Nov 02 23:23:06.948056 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:06.947624 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-818862-c954974c4-2vb2t" podUID="1160858d-2d42-445c-81f9-644fe3a81e5f" containerName="maas-api" containerID="cri-o://918cf39149830bbddc2ba2ea88a26301eb4d79f7892c3fa23d04ad289a3852bc" gracePeriod=30 Nov 02 23:23:07.223916 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:07.223813 5076 generic.go:356] "Generic (PLEG): container finished" podID="1160858d-2d42-445c-81f9-644fe3a81e5f" containerID="918cf39149830bbddc2ba2ea88a26301eb4d79f7892c3fa23d04ad289a3852bc" exitCode=0 Nov 02 23:23:07.224240 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:07.224206 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-818862-c954974c4-2vb2t" event={"ID":"1160858d-2d42-445c-81f9-644fe3a81e5f","Type":"ContainerDied","Data":"918cf39149830bbddc2ba2ea88a26301eb4d79f7892c3fa23d04ad289a3852bc"} Nov 02 23:23:07.291670 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:07.291642 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-818862-c954974c4-2vb2t" Nov 02 23:23:07.322929 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:07.322864 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1160858d-2d42-445c-81f9-644fe3a81e5f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1160858d-2d42-445c-81f9-644fe3a81e5f-maas-api-tls\") pod \"1160858d-2d42-445c-81f9-644fe3a81e5f\" (UID: \"1160858d-2d42-445c-81f9-644fe3a81e5f\") " Nov 02 23:23:07.323107 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:07.323080 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1160858d-2d42-445c-81f9-644fe3a81e5f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1160858d-2d42-445c-81f9-644fe3a81e5f-openshift-service-ca\") pod \"1160858d-2d42-445c-81f9-644fe3a81e5f\" (UID: \"1160858d-2d42-445c-81f9-644fe3a81e5f\") " Nov 02 23:23:07.323173 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:07.323136 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1160858d-2d42-445c-81f9-644fe3a81e5f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1160858d-2d42-445c-81f9-644fe3a81e5f-metrics-certs\") pod \"1160858d-2d42-445c-81f9-644fe3a81e5f\" (UID: \"1160858d-2d42-445c-81f9-644fe3a81e5f\") " Nov 02 23:23:07.323237 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:07.323220 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1160858d-2d42-445c-81f9-644fe3a81e5f-kube-api-access-p9whl\" (UniqueName: \"kubernetes.io/projected/1160858d-2d42-445c-81f9-644fe3a81e5f-kube-api-access-p9whl\") pod \"1160858d-2d42-445c-81f9-644fe3a81e5f\" (UID: \"1160858d-2d42-445c-81f9-644fe3a81e5f\") " Nov 02 23:23:07.323689 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:07.323642 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1160858d-2d42-445c-81f9-644fe3a81e5f-openshift-service-ca" pod "1160858d-2d42-445c-81f9-644fe3a81e5f" (UID: "1160858d-2d42-445c-81f9-644fe3a81e5f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:23:07.323867 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:07.323805 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1160858d-2d42-445c-81f9-644fe3a81e5f-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:07.325640 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:07.325608 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1160858d-2d42-445c-81f9-644fe3a81e5f-maas-api-tls" pod "1160858d-2d42-445c-81f9-644fe3a81e5f" (UID: "1160858d-2d42-445c-81f9-644fe3a81e5f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:23:07.326095 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:07.326061 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1160858d-2d42-445c-81f9-644fe3a81e5f-metrics-certs" pod "1160858d-2d42-445c-81f9-644fe3a81e5f" (UID: "1160858d-2d42-445c-81f9-644fe3a81e5f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:23:07.352155 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:07.345872 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1160858d-2d42-445c-81f9-644fe3a81e5f-kube-api-access-p9whl" pod "1160858d-2d42-445c-81f9-644fe3a81e5f" (UID: "1160858d-2d42-445c-81f9-644fe3a81e5f"). InnerVolumeSpecName "kube-api-access-p9whl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:23:07.430918 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:07.427488 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1160858d-2d42-445c-81f9-644fe3a81e5f-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:07.430918 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:07.427521 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9whl\" (UniqueName: \"kubernetes.io/projected/1160858d-2d42-445c-81f9-644fe3a81e5f-kube-api-access-p9whl\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:07.430918 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:07.427535 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1160858d-2d42-445c-81f9-644fe3a81e5f-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:08.231474 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:08.231447 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-818862-c954974c4-2vb2t" Nov 02 23:23:08.231969 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:08.231446 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-818862-c954974c4-2vb2t" event={"ID":"1160858d-2d42-445c-81f9-644fe3a81e5f","Type":"ContainerDied","Data":"e4c52aee2251b01e8f84aa0df7ca20b1789aa47e308edb44b3c58988a3abcab3"} Nov 02 23:23:08.231969 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:08.231586 5076 scope.go:122] "RemoveContainer" containerID="918cf39149830bbddc2ba2ea88a26301eb4d79f7892c3fa23d04ad289a3852bc" Nov 02 23:23:08.387702 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:08.387667 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-818862-c954974c4-2vb2t"] Nov 02 23:23:08.435247 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:08.435213 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-818862-c954974c4-2vb2t"] Nov 02 23:23:08.582418 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:08.582330 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt"] Nov 02 23:23:08.583043 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:08.583022 5076 cpu_manager.go:414] "containerMap: removing container" podUID="1160858d-2d42-445c-81f9-644fe3a81e5f" containerName="maas-api" Nov 02 23:23:08.583158 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:08.583044 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1160858d-2d42-445c-81f9-644fe3a81e5f" containerName="maas-api" Nov 02 23:23:08.583249 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:08.583235 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="1160858d-2d42-445c-81f9-644fe3a81e5f" containerName="maas-api" Nov 02 23:23:08.588250 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:08.588224 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt" Nov 02 23:23:08.637653 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:08.637623 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-c0064f-serving-cert\"" Nov 02 23:23:08.639799 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:08.639773 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt"] Nov 02 23:23:08.646528 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:08.646499 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cfd13c3-8704-4922-9a64-85882311360f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt\" (UID: \"0cfd13c3-8704-4922-9a64-85882311360f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt" Nov 02 23:23:08.646661 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:08.646545 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cfd13c3-8704-4922-9a64-85882311360f-metrics-certs\") pod \"maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt\" (UID: \"0cfd13c3-8704-4922-9a64-85882311360f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt" Nov 02 23:23:08.646745 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:08.646721 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc78b\" (UniqueName: \"kubernetes.io/projected/0cfd13c3-8704-4922-9a64-85882311360f-kube-api-access-vc78b\") pod \"maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt\" (UID: \"0cfd13c3-8704-4922-9a64-85882311360f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt" Nov 02 23:23:08.646941 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:08.646919 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cfd13c3-8704-4922-9a64-85882311360f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt\" (UID: \"0cfd13c3-8704-4922-9a64-85882311360f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt" Nov 02 23:23:08.748527 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:08.748496 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cfd13c3-8704-4922-9a64-85882311360f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt\" (UID: \"0cfd13c3-8704-4922-9a64-85882311360f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt" Nov 02 23:23:08.748708 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:08.748620 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cfd13c3-8704-4922-9a64-85882311360f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt\" (UID: \"0cfd13c3-8704-4922-9a64-85882311360f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt" Nov 02 23:23:08.748708 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:08.748650 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cfd13c3-8704-4922-9a64-85882311360f-metrics-certs\") pod \"maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt\" (UID: \"0cfd13c3-8704-4922-9a64-85882311360f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt" Nov 02 23:23:08.748708 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:23:08.748658 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0064f-serving-cert: secret "maas-api-e2e-ipp-b-c0064f-serving-cert" not found Nov 02 23:23:08.748904 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:23:08.748728 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cfd13c3-8704-4922-9a64-85882311360f-maas-api-tls podName:0cfd13c3-8704-4922-9a64-85882311360f nodeName:}" failed. No retries permitted until 2026-11-02 23:23:09.248705756 +0000 UTC m=+1838.217237535 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0cfd13c3-8704-4922-9a64-85882311360f-maas-api-tls") pod "maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt" (UID: "0cfd13c3-8704-4922-9a64-85882311360f") : secret "maas-api-e2e-ipp-b-c0064f-serving-cert" not found Nov 02 23:23:08.748904 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:08.748768 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc78b\" (UniqueName: \"kubernetes.io/projected/0cfd13c3-8704-4922-9a64-85882311360f-kube-api-access-vc78b\") pod \"maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt\" (UID: \"0cfd13c3-8704-4922-9a64-85882311360f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt" Nov 02 23:23:08.749478 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:08.749452 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cfd13c3-8704-4922-9a64-85882311360f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt\" (UID: \"0cfd13c3-8704-4922-9a64-85882311360f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt" Nov 02 23:23:08.753613 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:08.753586 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cfd13c3-8704-4922-9a64-85882311360f-metrics-certs\") pod \"maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt\" (UID: \"0cfd13c3-8704-4922-9a64-85882311360f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt" Nov 02 23:23:08.806540 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:08.806512 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc78b\" (UniqueName: \"kubernetes.io/projected/0cfd13c3-8704-4922-9a64-85882311360f-kube-api-access-vc78b\") pod \"maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt\" (UID: \"0cfd13c3-8704-4922-9a64-85882311360f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt" Nov 02 23:23:09.148185 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:09.148154 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1160858d-2d42-445c-81f9-644fe3a81e5f" path="/var/lib/kubelet/pods/1160858d-2d42-445c-81f9-644fe3a81e5f/volumes" Nov 02 23:23:09.256606 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:09.256576 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cfd13c3-8704-4922-9a64-85882311360f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt\" (UID: \"0cfd13c3-8704-4922-9a64-85882311360f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt" Nov 02 23:23:09.259023 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:09.258994 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cfd13c3-8704-4922-9a64-85882311360f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt\" (UID: \"0cfd13c3-8704-4922-9a64-85882311360f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt" Nov 02 23:23:09.502249 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:09.502219 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt" Nov 02 23:23:09.881679 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:09.881636 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt"] Nov 02 23:23:09.882426 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:23:09.882383 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cfd13c3_8704_4922_9a64_85882311360f.slice/crio-93eb67247138704c7476a20cbfd9073ca0a0a2c6d9f8c16293415d7c92caca65 WatchSource:0}: Error finding container 93eb67247138704c7476a20cbfd9073ca0a0a2c6d9f8c16293415d7c92caca65: Status 404 returned error can't find the container with id 93eb67247138704c7476a20cbfd9073ca0a0a2c6d9f8c16293415d7c92caca65 Nov 02 23:23:09.887107 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:23:09.887080 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:23:10.245836 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:10.245799 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt" event={"ID":"0cfd13c3-8704-4922-9a64-85882311360f","Type":"ContainerStarted","Data":"93eb67247138704c7476a20cbfd9073ca0a0a2c6d9f8c16293415d7c92caca65"} Nov 02 23:23:11.254317 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:11.254279 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt" event={"ID":"0cfd13c3-8704-4922-9a64-85882311360f","Type":"ContainerStarted","Data":"b9002a29f53a29d7fcfa2e61728b0cc96c34a2b61bfc1be1ba9d396f0667ab02"} Nov 02 23:23:11.254807 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:11.254343 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt" Nov 02 23:23:11.363726 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:11.363674 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt" podStartSLOduration=2.500600834 podStartE2EDuration="3.363661177s" totalImagesPullingTime="863.060343ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:23:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:23:11.363120481 +0000 UTC m=+1840.331652265" watchObservedRunningTime="2026-11-02 23:23:11.363661177 +0000 UTC m=+1840.332192960" Nov 02 23:23:11.766931 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:11.766882 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946"] Nov 02 23:23:11.767293 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:11.767242 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" podUID="9425db69-12a7-480d-b9a5-ef92b281a8ff" containerName="istio-proxy" containerID="cri-o://94c9aac842b981b9662952e1f36ac69f84e703cf320bdb80a877f5777c999110" gracePeriod=30 Nov 02 23:23:13.481216 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:13.481172 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp"] Nov 02 23:23:13.485935 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:13.485910 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" Nov 02 23:23:13.544188 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:13.544143 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86a56d46-242b-4863-8b05-4c5054fa0d90-istio-envoy\") pod \"e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp\" (UID: \"86a56d46-242b-4863-8b05-4c5054fa0d90\") " pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" Nov 02 23:23:13.544505 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:13.544480 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86a56d46-242b-4863-8b05-4c5054fa0d90-istiod-ca-cert\") pod \"e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp\" (UID: \"86a56d46-242b-4863-8b05-4c5054fa0d90\") " pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" Nov 02 23:23:13.544652 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:13.544528 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wssw\" (UniqueName: \"kubernetes.io/projected/86a56d46-242b-4863-8b05-4c5054fa0d90-kube-api-access-9wssw\") pod \"e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp\" (UID: \"86a56d46-242b-4863-8b05-4c5054fa0d90\") " pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" Nov 02 23:23:13.544652 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:13.544557 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86a56d46-242b-4863-8b05-4c5054fa0d90-istio-data\") pod \"e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp\" (UID: \"86a56d46-242b-4863-8b05-4c5054fa0d90\") " pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" Nov 02 23:23:13.544652 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:13.544610 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86a56d46-242b-4863-8b05-4c5054fa0d90-istio-token\") pod \"e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp\" (UID: \"86a56d46-242b-4863-8b05-4c5054fa0d90\") " pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" Nov 02 23:23:13.544824 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:13.544661 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86a56d46-242b-4863-8b05-4c5054fa0d90-istio-podinfo\") pod \"e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp\" (UID: \"86a56d46-242b-4863-8b05-4c5054fa0d90\") " pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" Nov 02 23:23:13.544824 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:13.544681 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp"] Nov 02 23:23:13.544824 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:13.544695 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a56d46-242b-4863-8b05-4c5054fa0d90-credential-socket\") pod \"e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp\" (UID: \"86a56d46-242b-4863-8b05-4c5054fa0d90\") " pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" Nov 02 23:23:13.545034 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:13.544925 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a56d46-242b-4863-8b05-4c5054fa0d90-workload-socket\") pod \"e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp\" (UID: \"86a56d46-242b-4863-8b05-4c5054fa0d90\") " pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" Nov 02 23:23:13.545034 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:13.544967 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86a56d46-242b-4863-8b05-4c5054fa0d90-workload-certs\") pod \"e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp\" (UID: \"86a56d46-242b-4863-8b05-4c5054fa0d90\") " pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" Nov 02 23:23:13.586347 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:13.586310 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-981724-openshift-default-dockercfg-2plzb\"" Nov 02 23:23:13.655912 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:13.646934 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86a56d46-242b-4863-8b05-4c5054fa0d90-workload-certs\") pod \"e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp\" (UID: \"86a56d46-242b-4863-8b05-4c5054fa0d90\") " pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" Nov 02 23:23:13.655912 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:13.647087 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86a56d46-242b-4863-8b05-4c5054fa0d90-istio-envoy\") pod \"e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp\" (UID: \"86a56d46-242b-4863-8b05-4c5054fa0d90\") " pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" Nov 02 23:23:13.655912 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:13.647138 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86a56d46-242b-4863-8b05-4c5054fa0d90-istiod-ca-cert\") pod \"e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp\" (UID: \"86a56d46-242b-4863-8b05-4c5054fa0d90\") " pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" Nov 02 23:23:13.655912 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:13.647179 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wssw\" (UniqueName: \"kubernetes.io/projected/86a56d46-242b-4863-8b05-4c5054fa0d90-kube-api-access-9wssw\") pod \"e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp\" (UID: \"86a56d46-242b-4863-8b05-4c5054fa0d90\") " pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" Nov 02 23:23:13.655912 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:13.647202 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86a56d46-242b-4863-8b05-4c5054fa0d90-istio-data\") pod \"e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp\" (UID: \"86a56d46-242b-4863-8b05-4c5054fa0d90\") " pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" Nov 02 23:23:13.655912 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:13.647255 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86a56d46-242b-4863-8b05-4c5054fa0d90-istio-token\") pod \"e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp\" (UID: \"86a56d46-242b-4863-8b05-4c5054fa0d90\") " pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" Nov 02 23:23:13.655912 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:13.647306 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86a56d46-242b-4863-8b05-4c5054fa0d90-istio-podinfo\") pod \"e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp\" (UID: \"86a56d46-242b-4863-8b05-4c5054fa0d90\") " pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" Nov 02 23:23:13.655912 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:13.647343 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a56d46-242b-4863-8b05-4c5054fa0d90-credential-socket\") pod \"e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp\" (UID: \"86a56d46-242b-4863-8b05-4c5054fa0d90\") " pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" Nov 02 23:23:13.655912 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:13.647458 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a56d46-242b-4863-8b05-4c5054fa0d90-workload-socket\") pod \"e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp\" (UID: \"86a56d46-242b-4863-8b05-4c5054fa0d90\") " pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" Nov 02 23:23:13.655912 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:13.648063 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a56d46-242b-4863-8b05-4c5054fa0d90-workload-socket\") pod \"e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp\" (UID: \"86a56d46-242b-4863-8b05-4c5054fa0d90\") " pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" Nov 02 23:23:13.655912 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:13.648359 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86a56d46-242b-4863-8b05-4c5054fa0d90-workload-certs\") pod \"e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp\" (UID: \"86a56d46-242b-4863-8b05-4c5054fa0d90\") " pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" Nov 02 23:23:13.655912 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:13.655647 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86a56d46-242b-4863-8b05-4c5054fa0d90-istio-envoy\") pod \"e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp\" (UID: \"86a56d46-242b-4863-8b05-4c5054fa0d90\") " pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" Nov 02 23:23:13.656901 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:13.656380 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86a56d46-242b-4863-8b05-4c5054fa0d90-istiod-ca-cert\") pod \"e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp\" (UID: \"86a56d46-242b-4863-8b05-4c5054fa0d90\") " pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" Nov 02 23:23:13.656975 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:13.656934 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86a56d46-242b-4863-8b05-4c5054fa0d90-istio-data\") pod \"e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp\" (UID: \"86a56d46-242b-4863-8b05-4c5054fa0d90\") " pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" Nov 02 23:23:13.695049 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:13.695012 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a56d46-242b-4863-8b05-4c5054fa0d90-credential-socket\") pod \"e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp\" (UID: \"86a56d46-242b-4863-8b05-4c5054fa0d90\") " pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" Nov 02 23:23:13.696355 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:13.696247 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86a56d46-242b-4863-8b05-4c5054fa0d90-istio-podinfo\") pod \"e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp\" (UID: \"86a56d46-242b-4863-8b05-4c5054fa0d90\") " pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" Nov 02 23:23:13.712957 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:13.712924 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86a56d46-242b-4863-8b05-4c5054fa0d90-istio-token\") pod \"e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp\" (UID: \"86a56d46-242b-4863-8b05-4c5054fa0d90\") " pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" Nov 02 23:23:13.713098 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:13.713018 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wssw\" (UniqueName: \"kubernetes.io/projected/86a56d46-242b-4863-8b05-4c5054fa0d90-kube-api-access-9wssw\") pod \"e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp\" (UID: \"86a56d46-242b-4863-8b05-4c5054fa0d90\") " pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" Nov 02 23:23:13.800179 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:13.800098 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" Nov 02 23:23:14.221931 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:14.221862 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp"] Nov 02 23:23:14.222086 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:23:14.222041 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86a56d46_242b_4863_8b05_4c5054fa0d90.slice/crio-1dffe92e8bb3493427a30c07b0e6aba77f683c95b7ea23d3cecb96ed77dad895 WatchSource:0}: Error finding container 1dffe92e8bb3493427a30c07b0e6aba77f683c95b7ea23d3cecb96ed77dad895: Status 404 returned error can't find the container with id 1dffe92e8bb3493427a30c07b0e6aba77f683c95b7ea23d3cecb96ed77dad895 Nov 02 23:23:14.224759 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:14.224725 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:23:14.224856 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:14.224793 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:23:14.224856 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:14.224821 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:23:14.302275 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:14.302246 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" event={"ID":"86a56d46-242b-4863-8b05-4c5054fa0d90","Type":"ContainerStarted","Data":"e636865b78586e81385059a5a941351a8e90000c6b818967049bdcd15cceb128"} Nov 02 23:23:14.302384 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:14.302305 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" event={"ID":"86a56d46-242b-4863-8b05-4c5054fa0d90","Type":"ContainerStarted","Data":"1dffe92e8bb3493427a30c07b0e6aba77f683c95b7ea23d3cecb96ed77dad895"} Nov 02 23:23:14.800880 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:14.800847 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" Nov 02 23:23:15.806169 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:15.806137 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" Nov 02 23:23:15.968337 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:15.968283 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" podStartSLOduration=2.968268385 podStartE2EDuration="2.968268385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:23:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:23:14.433570692 +0000 UTC m=+1843.402102489" watchObservedRunningTime="2026-11-02 23:23:15.968268385 +0000 UTC m=+1844.936800169" Nov 02 23:23:16.315027 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:16.314997 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" Nov 02 23:23:16.315929 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:16.315911 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" Nov 02 23:23:17.077950 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.077879 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" Nov 02 23:23:17.202241 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.202203 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9425db69-12a7-480d-b9a5-ef92b281a8ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9425db69-12a7-480d-b9a5-ef92b281a8ff-istio-data\") pod \"9425db69-12a7-480d-b9a5-ef92b281a8ff\" (UID: \"9425db69-12a7-480d-b9a5-ef92b281a8ff\") " Nov 02 23:23:17.202440 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.202283 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9425db69-12a7-480d-b9a5-ef92b281a8ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9425db69-12a7-480d-b9a5-ef92b281a8ff-credential-socket\") pod \"9425db69-12a7-480d-b9a5-ef92b281a8ff\" (UID: \"9425db69-12a7-480d-b9a5-ef92b281a8ff\") " Nov 02 23:23:17.202440 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.202333 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9425db69-12a7-480d-b9a5-ef92b281a8ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9425db69-12a7-480d-b9a5-ef92b281a8ff-istio-podinfo\") pod \"9425db69-12a7-480d-b9a5-ef92b281a8ff\" (UID: \"9425db69-12a7-480d-b9a5-ef92b281a8ff\") " Nov 02 23:23:17.202440 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.202396 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9425db69-12a7-480d-b9a5-ef92b281a8ff-istio-token\" (UniqueName: \"kubernetes.io/projected/9425db69-12a7-480d-b9a5-ef92b281a8ff-istio-token\") pod \"9425db69-12a7-480d-b9a5-ef92b281a8ff\" (UID: \"9425db69-12a7-480d-b9a5-ef92b281a8ff\") " Nov 02 23:23:17.202625 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.202497 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9425db69-12a7-480d-b9a5-ef92b281a8ff-istio-data" pod "9425db69-12a7-480d-b9a5-ef92b281a8ff" (UID: "9425db69-12a7-480d-b9a5-ef92b281a8ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:23:17.202625 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.202542 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9425db69-12a7-480d-b9a5-ef92b281a8ff-credential-socket" pod "9425db69-12a7-480d-b9a5-ef92b281a8ff" (UID: "9425db69-12a7-480d-b9a5-ef92b281a8ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:23:17.202954 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.202850 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9425db69-12a7-480d-b9a5-ef92b281a8ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9425db69-12a7-480d-b9a5-ef92b281a8ff-workload-socket\") pod \"9425db69-12a7-480d-b9a5-ef92b281a8ff\" (UID: \"9425db69-12a7-480d-b9a5-ef92b281a8ff\") " Nov 02 23:23:17.202954 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.202951 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9425db69-12a7-480d-b9a5-ef92b281a8ff-kube-api-access-plrkh\" (UniqueName: \"kubernetes.io/projected/9425db69-12a7-480d-b9a5-ef92b281a8ff-kube-api-access-plrkh\") pod \"9425db69-12a7-480d-b9a5-ef92b281a8ff\" (UID: \"9425db69-12a7-480d-b9a5-ef92b281a8ff\") " Nov 02 23:23:17.203113 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.203007 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9425db69-12a7-480d-b9a5-ef92b281a8ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9425db69-12a7-480d-b9a5-ef92b281a8ff-workload-certs\") pod \"9425db69-12a7-480d-b9a5-ef92b281a8ff\" (UID: \"9425db69-12a7-480d-b9a5-ef92b281a8ff\") " Nov 02 23:23:17.203179 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.203157 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9425db69-12a7-480d-b9a5-ef92b281a8ff-workload-socket" pod "9425db69-12a7-480d-b9a5-ef92b281a8ff" (UID: "9425db69-12a7-480d-b9a5-ef92b281a8ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:23:17.203265 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.203246 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9425db69-12a7-480d-b9a5-ef92b281a8ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9425db69-12a7-480d-b9a5-ef92b281a8ff-istiod-ca-cert\") pod \"9425db69-12a7-480d-b9a5-ef92b281a8ff\" (UID: \"9425db69-12a7-480d-b9a5-ef92b281a8ff\") " Nov 02 23:23:17.203368 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.203355 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9425db69-12a7-480d-b9a5-ef92b281a8ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9425db69-12a7-480d-b9a5-ef92b281a8ff-istio-envoy\") pod \"9425db69-12a7-480d-b9a5-ef92b281a8ff\" (UID: \"9425db69-12a7-480d-b9a5-ef92b281a8ff\") " Nov 02 23:23:17.204172 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.203679 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9425db69-12a7-480d-b9a5-ef92b281a8ff-workload-certs" pod "9425db69-12a7-480d-b9a5-ef92b281a8ff" (UID: "9425db69-12a7-480d-b9a5-ef92b281a8ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:23:17.204172 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.203905 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9425db69-12a7-480d-b9a5-ef92b281a8ff-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:17.204172 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.203927 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9425db69-12a7-480d-b9a5-ef92b281a8ff-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:17.204172 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.203942 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9425db69-12a7-480d-b9a5-ef92b281a8ff-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:17.204172 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.203957 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9425db69-12a7-480d-b9a5-ef92b281a8ff-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:17.204172 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.203990 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9425db69-12a7-480d-b9a5-ef92b281a8ff-istiod-ca-cert" pod "9425db69-12a7-480d-b9a5-ef92b281a8ff" (UID: "9425db69-12a7-480d-b9a5-ef92b281a8ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:23:17.205596 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.205325 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9425db69-12a7-480d-b9a5-ef92b281a8ff-istio-token" pod "9425db69-12a7-480d-b9a5-ef92b281a8ff" (UID: "9425db69-12a7-480d-b9a5-ef92b281a8ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:23:17.206108 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.206078 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9425db69-12a7-480d-b9a5-ef92b281a8ff-kube-api-access-plrkh" pod "9425db69-12a7-480d-b9a5-ef92b281a8ff" (UID: "9425db69-12a7-480d-b9a5-ef92b281a8ff"). InnerVolumeSpecName "kube-api-access-plrkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:23:17.206212 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.206129 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9425db69-12a7-480d-b9a5-ef92b281a8ff-istio-envoy" pod "9425db69-12a7-480d-b9a5-ef92b281a8ff" (UID: "9425db69-12a7-480d-b9a5-ef92b281a8ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:23:17.206434 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.206404 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9425db69-12a7-480d-b9a5-ef92b281a8ff-istio-podinfo" pod "9425db69-12a7-480d-b9a5-ef92b281a8ff" (UID: "9425db69-12a7-480d-b9a5-ef92b281a8ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:23:17.266917 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.266860 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt" Nov 02 23:23:17.305059 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.305025 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9425db69-12a7-480d-b9a5-ef92b281a8ff-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:17.305059 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.305054 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9425db69-12a7-480d-b9a5-ef92b281a8ff-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:17.305059 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.305065 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9425db69-12a7-480d-b9a5-ef92b281a8ff-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:17.305292 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.305074 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plrkh\" (UniqueName: \"kubernetes.io/projected/9425db69-12a7-480d-b9a5-ef92b281a8ff-kube-api-access-plrkh\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:17.305292 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.305084 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9425db69-12a7-480d-b9a5-ef92b281a8ff-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:17.325655 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.325615 5076 generic.go:356] "Generic (PLEG): container finished" podID="9425db69-12a7-480d-b9a5-ef92b281a8ff" containerID="94c9aac842b981b9662952e1f36ac69f84e703cf320bdb80a877f5777c999110" exitCode=0 Nov 02 23:23:17.325847 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.325675 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" Nov 02 23:23:17.325847 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.325700 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" event={"ID":"9425db69-12a7-480d-b9a5-ef92b281a8ff","Type":"ContainerDied","Data":"94c9aac842b981b9662952e1f36ac69f84e703cf320bdb80a877f5777c999110"} Nov 02 23:23:17.325847 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.325770 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946" event={"ID":"9425db69-12a7-480d-b9a5-ef92b281a8ff","Type":"ContainerDied","Data":"6d29fff5b720f795a40ffe84fb0f6239fe3c40d79a367229718d47079f600f1c"} Nov 02 23:23:17.325847 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.325810 5076 scope.go:122] "RemoveContainer" containerID="94c9aac842b981b9662952e1f36ac69f84e703cf320bdb80a877f5777c999110" Nov 02 23:23:17.337678 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.337653 5076 scope.go:122] "RemoveContainer" containerID="701de5b7736526c41ee427ab87c782202f12e81d4468487a14a205faa9b5ebe0" Nov 02 23:23:17.347861 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.347845 5076 scope.go:122] "RemoveContainer" containerID="94c9aac842b981b9662952e1f36ac69f84e703cf320bdb80a877f5777c999110" Nov 02 23:23:17.348171 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:23:17.348151 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94c9aac842b981b9662952e1f36ac69f84e703cf320bdb80a877f5777c999110\": container with ID starting with 94c9aac842b981b9662952e1f36ac69f84e703cf320bdb80a877f5777c999110 not found: ID does not exist" containerID="94c9aac842b981b9662952e1f36ac69f84e703cf320bdb80a877f5777c999110" Nov 02 23:23:17.348234 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.348178 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94c9aac842b981b9662952e1f36ac69f84e703cf320bdb80a877f5777c999110"} err="failed to get container status \"94c9aac842b981b9662952e1f36ac69f84e703cf320bdb80a877f5777c999110\": rpc error: code = NotFound desc = could not find container \"94c9aac842b981b9662952e1f36ac69f84e703cf320bdb80a877f5777c999110\": container with ID starting with 94c9aac842b981b9662952e1f36ac69f84e703cf320bdb80a877f5777c999110 not found: ID does not exist" Nov 02 23:23:17.348234 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.348196 5076 scope.go:122] "RemoveContainer" containerID="701de5b7736526c41ee427ab87c782202f12e81d4468487a14a205faa9b5ebe0" Nov 02 23:23:17.348429 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:23:17.348410 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"701de5b7736526c41ee427ab87c782202f12e81d4468487a14a205faa9b5ebe0\": container with ID starting with 701de5b7736526c41ee427ab87c782202f12e81d4468487a14a205faa9b5ebe0 not found: ID does not exist" containerID="701de5b7736526c41ee427ab87c782202f12e81d4468487a14a205faa9b5ebe0" Nov 02 23:23:17.348473 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.348435 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"701de5b7736526c41ee427ab87c782202f12e81d4468487a14a205faa9b5ebe0"} err="failed to get container status \"701de5b7736526c41ee427ab87c782202f12e81d4468487a14a205faa9b5ebe0\": rpc error: code = NotFound desc = could not find container \"701de5b7736526c41ee427ab87c782202f12e81d4468487a14a205faa9b5ebe0\": container with ID starting with 701de5b7736526c41ee427ab87c782202f12e81d4468487a14a205faa9b5ebe0 not found: ID does not exist" Nov 02 23:23:17.487780 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.487745 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946"] Nov 02 23:23:17.494480 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:17.494416 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-818862-openshift-default-667cf76c7-v8946"] Nov 02 23:23:19.148385 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:19.148352 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9425db69-12a7-480d-b9a5-ef92b281a8ff" path="/var/lib/kubelet/pods/9425db69-12a7-480d-b9a5-ef92b281a8ff/volumes" Nov 02 23:23:27.406206 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:27.406175 5076 generic.go:356] "Generic (PLEG): container finished" podID="a1632e0b-a576-4db6-ba82-dc11e89a7c00" containerID="12e662bb18e88e5acd93940191e506762a5c8d60d33f2c366c5d70e535342593" exitCode=0 Nov 02 23:23:27.406631 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:27.406248 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" event={"ID":"a1632e0b-a576-4db6-ba82-dc11e89a7c00","Type":"ContainerDied","Data":"12e662bb18e88e5acd93940191e506762a5c8d60d33f2c366c5d70e535342593"} Nov 02 23:23:27.406710 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:27.406696 5076 scope.go:122] "RemoveContainer" containerID="12e662bb18e88e5acd93940191e506762a5c8d60d33f2c366c5d70e535342593" Nov 02 23:23:27.408314 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:27.408288 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:23:27.408383 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:27.408353 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:23:27.408419 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:27.408386 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:23:28.437550 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:28.437505 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" event={"ID":"a1632e0b-a576-4db6-ba82-dc11e89a7c00","Type":"ContainerStarted","Data":"cdfd791b71e12d995d72723583b6310eb431817997838a103b2b84adac141348"} Nov 02 23:23:29.373949 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:29.373918 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" Nov 02 23:23:29.374119 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:29.374094 5076 patch_prober.go:29] interesting pod/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 02 23:23:29.374181 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:29.374163 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" podUID="a1632e0b-a576-4db6-ba82-dc11e89a7c00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 02 23:23:30.373985 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:30.373939 5076 patch_prober.go:29] interesting pod/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 02 23:23:30.374383 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:30.374007 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" podUID="a1632e0b-a576-4db6-ba82-dc11e89a7c00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 02 23:23:31.373147 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:31.373113 5076 patch_prober.go:29] interesting pod/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 02 23:23:31.373325 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:31.373166 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" podUID="a1632e0b-a576-4db6-ba82-dc11e89a7c00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 02 23:23:32.373923 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:32.373879 5076 patch_prober.go:29] interesting pod/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 02 23:23:32.374325 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:32.373952 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" podUID="a1632e0b-a576-4db6-ba82-dc11e89a7c00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 02 23:23:33.374067 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:33.374031 5076 patch_prober.go:29] interesting pod/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 02 23:23:33.374477 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:33.374081 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" podUID="a1632e0b-a576-4db6-ba82-dc11e89a7c00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 02 23:23:34.378049 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:34.378017 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" Nov 02 23:23:34.378449 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:34.378301 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" Nov 02 23:23:34.378873 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:34.378857 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" Nov 02 23:23:39.884634 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:23:39.884587 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:23:52.616376 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:52.616341 5076 generic.go:356] "Generic (PLEG): container finished" podID="c7bcd988-c4a5-46e6-bba1-c4346f21922d" containerID="3610146349edc6f65d4cd50dd26f467e09bb2d807ae0606d3e6dfd74516329f7" exitCode=0 Nov 02 23:23:52.616928 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:52.616416 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" event={"ID":"c7bcd988-c4a5-46e6-bba1-c4346f21922d","Type":"ContainerDied","Data":"3610146349edc6f65d4cd50dd26f467e09bb2d807ae0606d3e6dfd74516329f7"} Nov 02 23:23:52.616928 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:52.616815 5076 scope.go:122] "RemoveContainer" containerID="3610146349edc6f65d4cd50dd26f467e09bb2d807ae0606d3e6dfd74516329f7" Nov 02 23:23:52.618270 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:52.618238 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:23:52.618411 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:52.618312 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:23:52.618411 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:52.618349 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:23:53.640267 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:53.640222 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" event={"ID":"c7bcd988-c4a5-46e6-bba1-c4346f21922d","Type":"ContainerStarted","Data":"a7815162a8059e9cde8df60c7aa3ef764e862f6c0567ea5b14a0942c365491a4"} Nov 02 23:23:54.186966 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:54.186919 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" Nov 02 23:23:54.188283 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:54.188254 5076 patch_prober.go:29] interesting pod/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 23:23:54.188436 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:54.188324 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" podUID="c7bcd988-c4a5-46e6-bba1-c4346f21922d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 23:23:55.187423 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:55.187392 5076 patch_prober.go:29] interesting pod/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 23:23:55.187829 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:55.187439 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" podUID="c7bcd988-c4a5-46e6-bba1-c4346f21922d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 23:23:56.187538 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:56.187496 5076 patch_prober.go:29] interesting pod/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 23:23:56.188024 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:56.187564 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" podUID="c7bcd988-c4a5-46e6-bba1-c4346f21922d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 23:23:57.186878 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:57.186844 5076 patch_prober.go:29] interesting pod/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 23:23:57.187120 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:57.186940 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" podUID="c7bcd988-c4a5-46e6-bba1-c4346f21922d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 23:23:58.187751 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:58.187716 5076 patch_prober.go:29] interesting pod/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 23:23:58.188169 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:58.187780 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" podUID="c7bcd988-c4a5-46e6-bba1-c4346f21922d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 23:23:58.496151 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:58.496069 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5"] Nov 02 23:23:58.496702 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:58.496687 5076 cpu_manager.go:414] "containerMap: removing container" podUID="9425db69-12a7-480d-b9a5-ef92b281a8ff" containerName="istio-proxy" Nov 02 23:23:58.496745 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:58.496704 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9425db69-12a7-480d-b9a5-ef92b281a8ff" containerName="istio-proxy" Nov 02 23:23:58.496745 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:58.496715 5076 cpu_manager.go:414] "containerMap: removing container" podUID="9425db69-12a7-480d-b9a5-ef92b281a8ff" containerName="istio-proxy" Nov 02 23:23:58.496745 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:58.496720 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9425db69-12a7-480d-b9a5-ef92b281a8ff" containerName="istio-proxy" Nov 02 23:23:58.496848 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:58.496813 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="9425db69-12a7-480d-b9a5-ef92b281a8ff" containerName="istio-proxy" Nov 02 23:23:58.496848 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:58.496821 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="9425db69-12a7-480d-b9a5-ef92b281a8ff" containerName="istio-proxy" Nov 02 23:23:58.500537 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:58.500522 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5" Nov 02 23:23:58.554331 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:58.554296 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5"] Nov 02 23:23:58.597532 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:58.597497 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-9f5790-serving-cert\"" Nov 02 23:23:58.638461 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:58.638414 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv4fp\" (UniqueName: \"kubernetes.io/projected/d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc-kube-api-access-fv4fp\") pod \"maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5\" (UID: \"d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5" Nov 02 23:23:58.638658 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:58.638503 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc-metrics-certs\") pod \"maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5\" (UID: \"d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5" Nov 02 23:23:58.638658 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:58.638580 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5\" (UID: \"d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5" Nov 02 23:23:58.638803 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:58.638778 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5\" (UID: \"d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5" Nov 02 23:23:58.725490 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:58.719993 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" Nov 02 23:23:58.739945 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:58.739877 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc-metrics-certs\") pod \"maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5\" (UID: \"d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5" Nov 02 23:23:58.740118 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:58.739973 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5\" (UID: \"d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5" Nov 02 23:23:58.740178 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:23:58.740110 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5790-serving-cert: secret "maas-api-e2e-ipp-b-9f5790-serving-cert" not found Nov 02 23:23:58.740178 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:58.740158 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5\" (UID: \"d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5" Nov 02 23:23:58.740273 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:23:58.740196 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc-maas-api-tls podName:d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc nodeName:}" failed. No retries permitted until 2026-11-02 23:23:59.240175628 +0000 UTC m=+1888.208707406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc-maas-api-tls") pod "maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5" (UID: "d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc") : secret "maas-api-e2e-ipp-b-9f5790-serving-cert" not found Nov 02 23:23:58.740336 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:58.740284 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv4fp\" (UniqueName: \"kubernetes.io/projected/d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc-kube-api-access-fv4fp\") pod \"maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5\" (UID: \"d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5" Nov 02 23:23:58.740785 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:58.740760 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5\" (UID: \"d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5" Nov 02 23:23:58.742518 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:58.742499 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc-metrics-certs\") pod \"maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5\" (UID: \"d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5" Nov 02 23:23:58.821640 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:58.821561 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv4fp\" (UniqueName: \"kubernetes.io/projected/d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc-kube-api-access-fv4fp\") pod \"maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5\" (UID: \"d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5" Nov 02 23:23:59.192499 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:59.192471 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" Nov 02 23:23:59.251012 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:59.250975 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5\" (UID: \"d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5" Nov 02 23:23:59.253560 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:59.253526 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5\" (UID: \"d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5" Nov 02 23:23:59.437407 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:59.437372 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5" Nov 02 23:23:59.767978 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:59.760157 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" Nov 02 23:23:59.887985 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:59.887958 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5"] Nov 02 23:23:59.889210 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:23:59.889179 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9c92d5e_e50a_4e17_bc02_2b16c1bdb9dc.slice/crio-3a07ba2e6914487c578383b9d1ec090652f39ec180903d442338f8848d6d7bcf WatchSource:0}: Error finding container 3a07ba2e6914487c578383b9d1ec090652f39ec180903d442338f8848d6d7bcf: Status 404 returned error can't find the container with id 3a07ba2e6914487c578383b9d1ec090652f39ec180903d442338f8848d6d7bcf Nov 02 23:23:59.891016 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:23:59.890997 5076 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 23:24:00.758089 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:00.758057 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5" event={"ID":"d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc","Type":"ContainerStarted","Data":"3a07ba2e6914487c578383b9d1ec090652f39ec180903d442338f8848d6d7bcf"} Nov 02 23:24:01.771703 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:01.771662 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5" event={"ID":"d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc","Type":"ContainerStarted","Data":"bb1584d2e4a7ab4bb0430cb48642610e719da6bb7b584599bc70fbe2d43c16c9"} Nov 02 23:24:01.772231 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:01.772204 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5" Nov 02 23:24:01.896414 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:01.896363 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5" podStartSLOduration=2.80925104 podStartE2EDuration="3.896350445s" totalImagesPullingTime="1.087099405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:23:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:24:01.892969224 +0000 UTC m=+1890.861501008" watchObservedRunningTime="2026-11-02 23:24:01.896350445 +0000 UTC m=+1890.864882229" Nov 02 23:24:07.785368 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:07.785327 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5" Nov 02 23:24:09.884819 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:24:09.884774 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:24:10.837744 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:10.837652 5076 generic.go:356] "Generic (PLEG): container finished" podID="dfd27984-010f-441c-a3f4-553933c0529a" containerID="70c2c4ebc1f046f648bf9e4350e65bc41dfb010749d084db3dc8cff0ef6e30c4" exitCode=0 Nov 02 23:24:10.837744 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:10.837728 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" event={"ID":"dfd27984-010f-441c-a3f4-553933c0529a","Type":"ContainerDied","Data":"70c2c4ebc1f046f648bf9e4350e65bc41dfb010749d084db3dc8cff0ef6e30c4"} Nov 02 23:24:10.838221 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:10.838203 5076 scope.go:122] "RemoveContainer" containerID="70c2c4ebc1f046f648bf9e4350e65bc41dfb010749d084db3dc8cff0ef6e30c4" Nov 02 23:24:10.839709 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:10.839684 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:24:10.839784 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:10.839758 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:24:10.839833 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:10.839791 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:24:11.848768 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:11.848731 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" event={"ID":"dfd27984-010f-441c-a3f4-553933c0529a","Type":"ContainerStarted","Data":"db7259016578f9d9120ee7dc14f0d84c043ede86b683c463e84d7f64a4a5eb63"} Nov 02 23:24:12.149953 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:12.149850 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" Nov 02 23:24:12.151052 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:12.151026 5076 patch_prober.go:29] interesting pod/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 02 23:24:12.151182 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:12.151072 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" podUID="dfd27984-010f-441c-a3f4-553933c0529a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 02 23:24:12.951198 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:12.951154 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" Nov 02 23:24:13.150438 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:13.150409 5076 patch_prober.go:29] interesting pod/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 02 23:24:13.150634 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:13.150477 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" podUID="dfd27984-010f-441c-a3f4-553933c0529a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 02 23:24:14.149831 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:14.149791 5076 patch_prober.go:29] interesting pod/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 02 23:24:14.150319 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:14.149859 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" podUID="dfd27984-010f-441c-a3f4-553933c0529a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 02 23:24:15.150012 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:15.149975 5076 patch_prober.go:29] interesting pod/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 02 23:24:15.150420 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:15.150026 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" podUID="dfd27984-010f-441c-a3f4-553933c0529a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 02 23:24:16.149794 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:16.149760 5076 patch_prober.go:29] interesting pod/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 02 23:24:16.149998 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:16.149811 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" podUID="dfd27984-010f-441c-a3f4-553933c0529a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 02 23:24:17.155134 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:17.155104 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" Nov 02 23:24:17.156108 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:17.156086 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" Nov 02 23:24:18.716278 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:18.716240 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9"] Nov 02 23:24:18.716743 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:18.716441 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" podUID="5b6815fb-2006-4710-a166-db066f8359f8" containerName="istio-proxy" containerID="cri-o://2911d3e7565a9a9350d7419ecbcd958bf5fc23a82590df7e00f74e91e821e193" gracePeriod=30 Nov 02 23:24:21.880969 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:21.880933 5076 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:24:21.881383 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:21.880996 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" podUID="5b6815fb-2006-4710-a166-db066f8359f8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:24:22.782011 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:22.781979 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h"] Nov 02 23:24:22.782221 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:22.782193 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" podUID="db841b5b-fade-48e4-9515-ef923b888f42" containerName="istio-proxy" containerID="cri-o://50bebcfe439db99de3dd447638369c5a25e6b60507a29e399e952c04c69fadc9" gracePeriod=30 Nov 02 23:24:22.831355 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:22.831318 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn"] Nov 02 23:24:22.831579 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:22.831556 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" podUID="a67b3447-b8b2-4485-bacf-0d39a1352feb" containerName="istio-proxy" containerID="cri-o://6a8adf24919c12f4141a053259df0816b40eebbc8bfb7bdea94a8646b5711b41" gracePeriod=30 Nov 02 23:24:22.835116 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:22.835090 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8"] Nov 02 23:24:22.835300 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:22.835278 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" podUID="ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d" containerName="istio-proxy" containerID="cri-o://271338a4e786084b47c46ed1176d93ad1ca308b53f49763c1bb02adac63ab9d7" gracePeriod=30 Nov 02 23:24:22.882875 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:22.882837 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z"] Nov 02 23:24:22.883390 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:22.883101 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" podUID="9b820762-bd67-4d19-a050-224c6225fe67" containerName="istio-proxy" containerID="cri-o://7723c5dbb3f60e1d039437c32bc35fdf00bc8c48b4868e4db88417196782c66a" gracePeriod=30 Nov 02 23:24:23.953299 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:23.953263 5076 generic.go:356] "Generic (PLEG): container finished" podID="5b6815fb-2006-4710-a166-db066f8359f8" containerID="2911d3e7565a9a9350d7419ecbcd958bf5fc23a82590df7e00f74e91e821e193" exitCode=0 Nov 02 23:24:23.953700 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:23.953335 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" event={"ID":"5b6815fb-2006-4710-a166-db066f8359f8","Type":"ContainerDied","Data":"2911d3e7565a9a9350d7419ecbcd958bf5fc23a82590df7e00f74e91e821e193"} Nov 02 23:24:23.994399 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:23.994320 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" Nov 02 23:24:24.041498 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:24.041443 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b6815fb-2006-4710-a166-db066f8359f8-istio-token\" (UniqueName: \"kubernetes.io/projected/5b6815fb-2006-4710-a166-db066f8359f8-istio-token\") pod \"5b6815fb-2006-4710-a166-db066f8359f8\" (UID: \"5b6815fb-2006-4710-a166-db066f8359f8\") " Nov 02 23:24:24.041746 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:24.041572 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b6815fb-2006-4710-a166-db066f8359f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b6815fb-2006-4710-a166-db066f8359f8-istio-envoy\") pod \"5b6815fb-2006-4710-a166-db066f8359f8\" (UID: \"5b6815fb-2006-4710-a166-db066f8359f8\") " Nov 02 23:24:24.041746 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:24.041632 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b6815fb-2006-4710-a166-db066f8359f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b6815fb-2006-4710-a166-db066f8359f8-workload-certs\") pod \"5b6815fb-2006-4710-a166-db066f8359f8\" (UID: \"5b6815fb-2006-4710-a166-db066f8359f8\") " Nov 02 23:24:24.041746 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:24.041699 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b6815fb-2006-4710-a166-db066f8359f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b6815fb-2006-4710-a166-db066f8359f8-istio-podinfo\") pod \"5b6815fb-2006-4710-a166-db066f8359f8\" (UID: \"5b6815fb-2006-4710-a166-db066f8359f8\") " Nov 02 23:24:24.041746 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:24.041728 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b6815fb-2006-4710-a166-db066f8359f8-kube-api-access-xzbgc\" (UniqueName: \"kubernetes.io/projected/5b6815fb-2006-4710-a166-db066f8359f8-kube-api-access-xzbgc\") pod \"5b6815fb-2006-4710-a166-db066f8359f8\" (UID: \"5b6815fb-2006-4710-a166-db066f8359f8\") " Nov 02 23:24:24.042101 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:24.041765 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b6815fb-2006-4710-a166-db066f8359f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b6815fb-2006-4710-a166-db066f8359f8-istiod-ca-cert\") pod \"5b6815fb-2006-4710-a166-db066f8359f8\" (UID: \"5b6815fb-2006-4710-a166-db066f8359f8\") " Nov 02 23:24:24.042101 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:24.041823 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b6815fb-2006-4710-a166-db066f8359f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6815fb-2006-4710-a166-db066f8359f8-credential-socket\") pod \"5b6815fb-2006-4710-a166-db066f8359f8\" (UID: \"5b6815fb-2006-4710-a166-db066f8359f8\") " Nov 02 23:24:24.042101 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:24.041881 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b6815fb-2006-4710-a166-db066f8359f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b6815fb-2006-4710-a166-db066f8359f8-istio-data\") pod \"5b6815fb-2006-4710-a166-db066f8359f8\" (UID: \"5b6815fb-2006-4710-a166-db066f8359f8\") " Nov 02 23:24:24.042101 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:24.041938 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b6815fb-2006-4710-a166-db066f8359f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6815fb-2006-4710-a166-db066f8359f8-workload-socket\") pod \"5b6815fb-2006-4710-a166-db066f8359f8\" (UID: \"5b6815fb-2006-4710-a166-db066f8359f8\") " Nov 02 23:24:24.055409 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:24.055271 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6815fb-2006-4710-a166-db066f8359f8-workload-certs" pod "5b6815fb-2006-4710-a166-db066f8359f8" (UID: "5b6815fb-2006-4710-a166-db066f8359f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:24.056231 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:24.056192 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6815fb-2006-4710-a166-db066f8359f8-credential-socket" pod "5b6815fb-2006-4710-a166-db066f8359f8" (UID: "5b6815fb-2006-4710-a166-db066f8359f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:24.058635 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:24.058612 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6815fb-2006-4710-a166-db066f8359f8-istio-data" pod "5b6815fb-2006-4710-a166-db066f8359f8" (UID: "5b6815fb-2006-4710-a166-db066f8359f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:24.059353 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:24.059325 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b6815fb-2006-4710-a166-db066f8359f8-istiod-ca-cert" pod "5b6815fb-2006-4710-a166-db066f8359f8" (UID: "5b6815fb-2006-4710-a166-db066f8359f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:24:24.069506 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:24.069463 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6815fb-2006-4710-a166-db066f8359f8-workload-socket" pod "5b6815fb-2006-4710-a166-db066f8359f8" (UID: "5b6815fb-2006-4710-a166-db066f8359f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:24.071035 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:24.070988 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6815fb-2006-4710-a166-db066f8359f8-istio-envoy" pod "5b6815fb-2006-4710-a166-db066f8359f8" (UID: "5b6815fb-2006-4710-a166-db066f8359f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:24.071664 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:24.071624 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b6815fb-2006-4710-a166-db066f8359f8-istio-podinfo" pod "5b6815fb-2006-4710-a166-db066f8359f8" (UID: "5b6815fb-2006-4710-a166-db066f8359f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:24:24.072413 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:24.072371 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b6815fb-2006-4710-a166-db066f8359f8-istio-token" pod "5b6815fb-2006-4710-a166-db066f8359f8" (UID: "5b6815fb-2006-4710-a166-db066f8359f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:24:24.072413 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:24.072400 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b6815fb-2006-4710-a166-db066f8359f8-kube-api-access-xzbgc" pod "5b6815fb-2006-4710-a166-db066f8359f8" (UID: "5b6815fb-2006-4710-a166-db066f8359f8"). InnerVolumeSpecName "kube-api-access-xzbgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:24:24.151676 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:24.151643 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6815fb-2006-4710-a166-db066f8359f8-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:24.151949 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:24.151928 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b6815fb-2006-4710-a166-db066f8359f8-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:24.152056 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:24.151957 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6815fb-2006-4710-a166-db066f8359f8-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:24.152056 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:24.151972 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b6815fb-2006-4710-a166-db066f8359f8-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:24.152056 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:24.151985 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b6815fb-2006-4710-a166-db066f8359f8-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:24.152056 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:24.151998 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b6815fb-2006-4710-a166-db066f8359f8-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:24.152056 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:24.152012 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b6815fb-2006-4710-a166-db066f8359f8-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:24.152056 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:24.152025 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzbgc\" (UniqueName: \"kubernetes.io/projected/5b6815fb-2006-4710-a166-db066f8359f8-kube-api-access-xzbgc\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:24.152056 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:24.152040 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b6815fb-2006-4710-a166-db066f8359f8-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:24.960761 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:24.960737 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" Nov 02 23:24:24.961242 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:24.960736 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9" event={"ID":"5b6815fb-2006-4710-a166-db066f8359f8","Type":"ContainerDied","Data":"95b9dd08e556991b54f39d1769549e3c6a5c36eb25b264aa07368853caa4def3"} Nov 02 23:24:24.961242 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:24.960881 5076 scope.go:122] "RemoveContainer" containerID="2911d3e7565a9a9350d7419ecbcd958bf5fc23a82590df7e00f74e91e821e193" Nov 02 23:24:25.121713 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:25.121676 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9"] Nov 02 23:24:25.168697 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:25.168667 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8qdk9"] Nov 02 23:24:25.216310 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:25.216228 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:24:25.216310 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:25.216289 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" podUID="a67b3447-b8b2-4485-bacf-0d39a1352feb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:24:25.462763 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:25.462722 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:24:25.462989 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:25.462782 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" podUID="db841b5b-fade-48e4-9515-ef923b888f42" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:24:27.177301 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:27.177264 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b6815fb-2006-4710-a166-db066f8359f8" path="/var/lib/kubelet/pods/5b6815fb-2006-4710-a166-db066f8359f8/volumes" Nov 02 23:24:28.080324 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.080141 5076 generic.go:356] "Generic (PLEG): container finished" podID="db841b5b-fade-48e4-9515-ef923b888f42" containerID="50bebcfe439db99de3dd447638369c5a25e6b60507a29e399e952c04c69fadc9" exitCode=0 Nov 02 23:24:28.080324 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.080208 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" event={"ID":"db841b5b-fade-48e4-9515-ef923b888f42","Type":"ContainerDied","Data":"50bebcfe439db99de3dd447638369c5a25e6b60507a29e399e952c04c69fadc9"} Nov 02 23:24:28.082151 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.082126 5076 generic.go:356] "Generic (PLEG): container finished" podID="a67b3447-b8b2-4485-bacf-0d39a1352feb" containerID="6a8adf24919c12f4141a053259df0816b40eebbc8bfb7bdea94a8646b5711b41" exitCode=0 Nov 02 23:24:28.082309 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.082289 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" event={"ID":"a67b3447-b8b2-4485-bacf-0d39a1352feb","Type":"ContainerDied","Data":"6a8adf24919c12f4141a053259df0816b40eebbc8bfb7bdea94a8646b5711b41"} Nov 02 23:24:28.084426 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.084227 5076 generic.go:356] "Generic (PLEG): container finished" podID="9b820762-bd67-4d19-a050-224c6225fe67" containerID="7723c5dbb3f60e1d039437c32bc35fdf00bc8c48b4868e4db88417196782c66a" exitCode=0 Nov 02 23:24:28.084426 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.084363 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" event={"ID":"9b820762-bd67-4d19-a050-224c6225fe67","Type":"ContainerDied","Data":"7723c5dbb3f60e1d039437c32bc35fdf00bc8c48b4868e4db88417196782c66a"} Nov 02 23:24:28.086540 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.086457 5076 generic.go:356] "Generic (PLEG): container finished" podID="ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d" containerID="271338a4e786084b47c46ed1176d93ad1ca308b53f49763c1bb02adac63ab9d7" exitCode=0 Nov 02 23:24:28.086540 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.086490 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" event={"ID":"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d","Type":"ContainerDied","Data":"271338a4e786084b47c46ed1176d93ad1ca308b53f49763c1bb02adac63ab9d7"} Nov 02 23:24:28.120615 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.120590 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" Nov 02 23:24:28.152986 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.152956 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db841b5b-fade-48e4-9515-ef923b888f42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db841b5b-fade-48e4-9515-ef923b888f42-istiod-ca-cert\") pod \"db841b5b-fade-48e4-9515-ef923b888f42\" (UID: \"db841b5b-fade-48e4-9515-ef923b888f42\") " Nov 02 23:24:28.153145 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.153015 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db841b5b-fade-48e4-9515-ef923b888f42-istio-token\" (UniqueName: \"kubernetes.io/projected/db841b5b-fade-48e4-9515-ef923b888f42-istio-token\") pod \"db841b5b-fade-48e4-9515-ef923b888f42\" (UID: \"db841b5b-fade-48e4-9515-ef923b888f42\") " Nov 02 23:24:28.153145 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.153094 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db841b5b-fade-48e4-9515-ef923b888f42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db841b5b-fade-48e4-9515-ef923b888f42-istio-data\") pod \"db841b5b-fade-48e4-9515-ef923b888f42\" (UID: \"db841b5b-fade-48e4-9515-ef923b888f42\") " Nov 02 23:24:28.153268 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.153141 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db841b5b-fade-48e4-9515-ef923b888f42-kube-api-access-5l658\" (UniqueName: \"kubernetes.io/projected/db841b5b-fade-48e4-9515-ef923b888f42-kube-api-access-5l658\") pod \"db841b5b-fade-48e4-9515-ef923b888f42\" (UID: \"db841b5b-fade-48e4-9515-ef923b888f42\") " Nov 02 23:24:28.153268 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.153195 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db841b5b-fade-48e4-9515-ef923b888f42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db841b5b-fade-48e4-9515-ef923b888f42-istio-envoy\") pod \"db841b5b-fade-48e4-9515-ef923b888f42\" (UID: \"db841b5b-fade-48e4-9515-ef923b888f42\") " Nov 02 23:24:28.153372 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.153291 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db841b5b-fade-48e4-9515-ef923b888f42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db841b5b-fade-48e4-9515-ef923b888f42-workload-certs\") pod \"db841b5b-fade-48e4-9515-ef923b888f42\" (UID: \"db841b5b-fade-48e4-9515-ef923b888f42\") " Nov 02 23:24:28.153431 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.153392 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db841b5b-fade-48e4-9515-ef923b888f42-istiod-ca-cert" pod "db841b5b-fade-48e4-9515-ef923b888f42" (UID: "db841b5b-fade-48e4-9515-ef923b888f42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:24:28.153431 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.153421 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db841b5b-fade-48e4-9515-ef923b888f42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db841b5b-fade-48e4-9515-ef923b888f42-credential-socket\") pod \"db841b5b-fade-48e4-9515-ef923b888f42\" (UID: \"db841b5b-fade-48e4-9515-ef923b888f42\") " Nov 02 23:24:28.153538 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.153464 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db841b5b-fade-48e4-9515-ef923b888f42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db841b5b-fade-48e4-9515-ef923b888f42-istio-podinfo\") pod \"db841b5b-fade-48e4-9515-ef923b888f42\" (UID: \"db841b5b-fade-48e4-9515-ef923b888f42\") " Nov 02 23:24:28.153538 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.153520 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db841b5b-fade-48e4-9515-ef923b888f42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db841b5b-fade-48e4-9515-ef923b888f42-workload-socket\") pod \"db841b5b-fade-48e4-9515-ef923b888f42\" (UID: \"db841b5b-fade-48e4-9515-ef923b888f42\") " Nov 02 23:24:28.153718 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.153695 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db841b5b-fade-48e4-9515-ef923b888f42-workload-certs" pod "db841b5b-fade-48e4-9515-ef923b888f42" (UID: "db841b5b-fade-48e4-9515-ef923b888f42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:28.154154 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.154113 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db841b5b-fade-48e4-9515-ef923b888f42-workload-socket" pod "db841b5b-fade-48e4-9515-ef923b888f42" (UID: "db841b5b-fade-48e4-9515-ef923b888f42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:28.154283 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.154197 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db841b5b-fade-48e4-9515-ef923b888f42-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:28.154283 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.154219 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db841b5b-fade-48e4-9515-ef923b888f42-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:28.154283 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.154224 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db841b5b-fade-48e4-9515-ef923b888f42-credential-socket" pod "db841b5b-fade-48e4-9515-ef923b888f42" (UID: "db841b5b-fade-48e4-9515-ef923b888f42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:28.154586 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.154502 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db841b5b-fade-48e4-9515-ef923b888f42-istio-data" pod "db841b5b-fade-48e4-9515-ef923b888f42" (UID: "db841b5b-fade-48e4-9515-ef923b888f42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:28.155749 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.155714 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db841b5b-fade-48e4-9515-ef923b888f42-istio-token" pod "db841b5b-fade-48e4-9515-ef923b888f42" (UID: "db841b5b-fade-48e4-9515-ef923b888f42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:24:28.157187 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.157150 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db841b5b-fade-48e4-9515-ef923b888f42-istio-podinfo" pod "db841b5b-fade-48e4-9515-ef923b888f42" (UID: "db841b5b-fade-48e4-9515-ef923b888f42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:24:28.157311 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.157199 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db841b5b-fade-48e4-9515-ef923b888f42-istio-envoy" pod "db841b5b-fade-48e4-9515-ef923b888f42" (UID: "db841b5b-fade-48e4-9515-ef923b888f42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:28.158847 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.158820 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db841b5b-fade-48e4-9515-ef923b888f42-kube-api-access-5l658" pod "db841b5b-fade-48e4-9515-ef923b888f42" (UID: "db841b5b-fade-48e4-9515-ef923b888f42"). InnerVolumeSpecName "kube-api-access-5l658". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:24:28.225681 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.225653 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" Nov 02 23:24:28.240228 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.240202 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" Nov 02 23:24:28.247472 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.247447 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" Nov 02 23:24:28.254912 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.254872 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-istio-envoy\") pod \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\" (UID: \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\") " Nov 02 23:24:28.255014 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.254938 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b820762-bd67-4d19-a050-224c6225fe67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b820762-bd67-4d19-a050-224c6225fe67-istiod-ca-cert\") pod \"9b820762-bd67-4d19-a050-224c6225fe67\" (UID: \"9b820762-bd67-4d19-a050-224c6225fe67\") " Nov 02 23:24:28.255014 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.254997 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-istio-podinfo\") pod \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\" (UID: \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\") " Nov 02 23:24:28.255093 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.255018 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-istiod-ca-cert\") pod \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\" (UID: \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\") " Nov 02 23:24:28.255093 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.255046 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-credential-socket\") pod \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\" (UID: \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\") " Nov 02 23:24:28.255265 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.255245 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a67b3447-b8b2-4485-bacf-0d39a1352feb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a67b3447-b8b2-4485-bacf-0d39a1352feb-istio-data\") pod \"a67b3447-b8b2-4485-bacf-0d39a1352feb\" (UID: \"a67b3447-b8b2-4485-bacf-0d39a1352feb\") " Nov 02 23:24:28.255323 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.255301 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b820762-bd67-4d19-a050-224c6225fe67-istiod-ca-cert" pod "9b820762-bd67-4d19-a050-224c6225fe67" (UID: "9b820762-bd67-4d19-a050-224c6225fe67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:24:28.255381 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.255317 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-istio-data\") pod \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\" (UID: \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\") " Nov 02 23:24:28.255381 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.255338 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-credential-socket" pod "ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d" (UID: "ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:28.255381 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.255367 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-istio-token\" (UniqueName: \"kubernetes.io/projected/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-istio-token\") pod \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\" (UID: \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\") " Nov 02 23:24:28.255542 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.255395 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b820762-bd67-4d19-a050-224c6225fe67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b820762-bd67-4d19-a050-224c6225fe67-istio-data\") pod \"9b820762-bd67-4d19-a050-224c6225fe67\" (UID: \"9b820762-bd67-4d19-a050-224c6225fe67\") " Nov 02 23:24:28.255542 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.255428 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b820762-bd67-4d19-a050-224c6225fe67-kube-api-access-s8mr7\" (UniqueName: \"kubernetes.io/projected/9b820762-bd67-4d19-a050-224c6225fe67-kube-api-access-s8mr7\") pod \"9b820762-bd67-4d19-a050-224c6225fe67\" (UID: \"9b820762-bd67-4d19-a050-224c6225fe67\") " Nov 02 23:24:28.255542 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.255460 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a67b3447-b8b2-4485-bacf-0d39a1352feb-istio-token\" (UniqueName: \"kubernetes.io/projected/a67b3447-b8b2-4485-bacf-0d39a1352feb-istio-token\") pod \"a67b3447-b8b2-4485-bacf-0d39a1352feb\" (UID: \"a67b3447-b8b2-4485-bacf-0d39a1352feb\") " Nov 02 23:24:28.255542 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.255479 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67b3447-b8b2-4485-bacf-0d39a1352feb-istio-data" pod "a67b3447-b8b2-4485-bacf-0d39a1352feb" (UID: "a67b3447-b8b2-4485-bacf-0d39a1352feb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:28.255542 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.255490 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-kube-api-access-dg9g2\" (UniqueName: \"kubernetes.io/projected/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-kube-api-access-dg9g2\") pod \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\" (UID: \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\") " Nov 02 23:24:28.255542 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.255431 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-istiod-ca-cert" pod "ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d" (UID: "ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:24:28.255848 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.255552 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a67b3447-b8b2-4485-bacf-0d39a1352feb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a67b3447-b8b2-4485-bacf-0d39a1352feb-workload-certs\") pod \"a67b3447-b8b2-4485-bacf-0d39a1352feb\" (UID: \"a67b3447-b8b2-4485-bacf-0d39a1352feb\") " Nov 02 23:24:28.255848 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.255554 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-istio-data" pod "ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d" (UID: "ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:28.255848 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.255590 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-workload-certs\") pod \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\" (UID: \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\") " Nov 02 23:24:28.255848 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.255638 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a67b3447-b8b2-4485-bacf-0d39a1352feb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67b3447-b8b2-4485-bacf-0d39a1352feb-credential-socket\") pod \"a67b3447-b8b2-4485-bacf-0d39a1352feb\" (UID: \"a67b3447-b8b2-4485-bacf-0d39a1352feb\") " Nov 02 23:24:28.255848 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.255684 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-workload-socket\") pod \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\" (UID: \"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d\") " Nov 02 23:24:28.255848 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.255718 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b820762-bd67-4d19-a050-224c6225fe67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b820762-bd67-4d19-a050-224c6225fe67-istio-podinfo\") pod \"9b820762-bd67-4d19-a050-224c6225fe67\" (UID: \"9b820762-bd67-4d19-a050-224c6225fe67\") " Nov 02 23:24:28.255848 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.255768 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b820762-bd67-4d19-a050-224c6225fe67-istio-token\" (UniqueName: \"kubernetes.io/projected/9b820762-bd67-4d19-a050-224c6225fe67-istio-token\") pod \"9b820762-bd67-4d19-a050-224c6225fe67\" (UID: \"9b820762-bd67-4d19-a050-224c6225fe67\") " Nov 02 23:24:28.255848 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.255800 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a67b3447-b8b2-4485-bacf-0d39a1352feb-kube-api-access-n8wxz\" (UniqueName: \"kubernetes.io/projected/a67b3447-b8b2-4485-bacf-0d39a1352feb-kube-api-access-n8wxz\") pod \"a67b3447-b8b2-4485-bacf-0d39a1352feb\" (UID: \"a67b3447-b8b2-4485-bacf-0d39a1352feb\") " Nov 02 23:24:28.255848 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.255831 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b820762-bd67-4d19-a050-224c6225fe67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b820762-bd67-4d19-a050-224c6225fe67-istio-envoy\") pod \"9b820762-bd67-4d19-a050-224c6225fe67\" (UID: \"9b820762-bd67-4d19-a050-224c6225fe67\") " Nov 02 23:24:28.256366 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.255870 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b820762-bd67-4d19-a050-224c6225fe67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b820762-bd67-4d19-a050-224c6225fe67-workload-socket\") pod \"9b820762-bd67-4d19-a050-224c6225fe67\" (UID: \"9b820762-bd67-4d19-a050-224c6225fe67\") " Nov 02 23:24:28.256366 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.255939 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b820762-bd67-4d19-a050-224c6225fe67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b820762-bd67-4d19-a050-224c6225fe67-credential-socket\") pod \"9b820762-bd67-4d19-a050-224c6225fe67\" (UID: \"9b820762-bd67-4d19-a050-224c6225fe67\") " Nov 02 23:24:28.256366 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.255966 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a67b3447-b8b2-4485-bacf-0d39a1352feb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67b3447-b8b2-4485-bacf-0d39a1352feb-workload-socket\") pod \"a67b3447-b8b2-4485-bacf-0d39a1352feb\" (UID: \"a67b3447-b8b2-4485-bacf-0d39a1352feb\") " Nov 02 23:24:28.256366 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.256024 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b820762-bd67-4d19-a050-224c6225fe67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b820762-bd67-4d19-a050-224c6225fe67-workload-certs\") pod \"9b820762-bd67-4d19-a050-224c6225fe67\" (UID: \"9b820762-bd67-4d19-a050-224c6225fe67\") " Nov 02 23:24:28.256366 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.256027 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b820762-bd67-4d19-a050-224c6225fe67-istio-data" pod "9b820762-bd67-4d19-a050-224c6225fe67" (UID: "9b820762-bd67-4d19-a050-224c6225fe67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:28.256366 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.256101 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a67b3447-b8b2-4485-bacf-0d39a1352feb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a67b3447-b8b2-4485-bacf-0d39a1352feb-istio-envoy\") pod \"a67b3447-b8b2-4485-bacf-0d39a1352feb\" (UID: \"a67b3447-b8b2-4485-bacf-0d39a1352feb\") " Nov 02 23:24:28.256366 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.256226 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a67b3447-b8b2-4485-bacf-0d39a1352feb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a67b3447-b8b2-4485-bacf-0d39a1352feb-istiod-ca-cert\") pod \"a67b3447-b8b2-4485-bacf-0d39a1352feb\" (UID: \"a67b3447-b8b2-4485-bacf-0d39a1352feb\") " Nov 02 23:24:28.256366 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.256237 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b820762-bd67-4d19-a050-224c6225fe67-workload-certs" pod "9b820762-bd67-4d19-a050-224c6225fe67" (UID: "9b820762-bd67-4d19-a050-224c6225fe67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:28.256366 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.256268 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a67b3447-b8b2-4485-bacf-0d39a1352feb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a67b3447-b8b2-4485-bacf-0d39a1352feb-istio-podinfo\") pod \"a67b3447-b8b2-4485-bacf-0d39a1352feb\" (UID: \"a67b3447-b8b2-4485-bacf-0d39a1352feb\") " Nov 02 23:24:28.256812 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.256417 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67b3447-b8b2-4485-bacf-0d39a1352feb-workload-certs" pod "a67b3447-b8b2-4485-bacf-0d39a1352feb" (UID: "a67b3447-b8b2-4485-bacf-0d39a1352feb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:28.256812 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.256589 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-workload-certs" pod "ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d" (UID: "ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:28.257061 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.257041 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:28.257147 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.257069 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l658\" (UniqueName: \"kubernetes.io/projected/db841b5b-fade-48e4-9515-ef923b888f42-kube-api-access-5l658\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:28.257147 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.257084 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db841b5b-fade-48e4-9515-ef923b888f42-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:28.257147 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.257100 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b820762-bd67-4d19-a050-224c6225fe67-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:28.257147 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.257115 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b820762-bd67-4d19-a050-224c6225fe67-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:28.257147 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.257128 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db841b5b-fade-48e4-9515-ef923b888f42-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:28.257147 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.257142 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db841b5b-fade-48e4-9515-ef923b888f42-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:28.257482 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.257156 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:28.257482 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.257170 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db841b5b-fade-48e4-9515-ef923b888f42-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:28.257482 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.257184 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:28.257482 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.257199 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db841b5b-fade-48e4-9515-ef923b888f42-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:28.257482 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.257212 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a67b3447-b8b2-4485-bacf-0d39a1352feb-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:28.257482 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.257226 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:28.257482 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.257241 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b820762-bd67-4d19-a050-224c6225fe67-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:28.257482 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.257254 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db841b5b-fade-48e4-9515-ef923b888f42-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:28.257482 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.257267 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a67b3447-b8b2-4485-bacf-0d39a1352feb-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:28.258056 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.257485 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67b3447-b8b2-4485-bacf-0d39a1352feb-credential-socket" pod "a67b3447-b8b2-4485-bacf-0d39a1352feb" (UID: "a67b3447-b8b2-4485-bacf-0d39a1352feb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:28.258056 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.257686 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-workload-socket" pod "ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d" (UID: "ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:28.258767 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.257774 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b820762-bd67-4d19-a050-224c6225fe67-kube-api-access-s8mr7" pod "9b820762-bd67-4d19-a050-224c6225fe67" (UID: "9b820762-bd67-4d19-a050-224c6225fe67"). InnerVolumeSpecName "kube-api-access-s8mr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:24:28.258767 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.257913 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b820762-bd67-4d19-a050-224c6225fe67-credential-socket" pod "9b820762-bd67-4d19-a050-224c6225fe67" (UID: "9b820762-bd67-4d19-a050-224c6225fe67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:28.258767 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.257971 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67b3447-b8b2-4485-bacf-0d39a1352feb-workload-socket" pod "a67b3447-b8b2-4485-bacf-0d39a1352feb" (UID: "a67b3447-b8b2-4485-bacf-0d39a1352feb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:28.258767 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.258062 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-istio-podinfo" pod "ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d" (UID: "ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:24:28.258767 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.258122 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a67b3447-b8b2-4485-bacf-0d39a1352feb-istio-token" pod "a67b3447-b8b2-4485-bacf-0d39a1352feb" (UID: "a67b3447-b8b2-4485-bacf-0d39a1352feb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:24:28.258767 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.258305 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a67b3447-b8b2-4485-bacf-0d39a1352feb-istiod-ca-cert" pod "a67b3447-b8b2-4485-bacf-0d39a1352feb" (UID: "a67b3447-b8b2-4485-bacf-0d39a1352feb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:24:28.258767 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.258361 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-kube-api-access-dg9g2" pod "ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d" (UID: "ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d"). InnerVolumeSpecName "kube-api-access-dg9g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:24:28.258767 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.258758 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b820762-bd67-4d19-a050-224c6225fe67-workload-socket" pod "9b820762-bd67-4d19-a050-224c6225fe67" (UID: "9b820762-bd67-4d19-a050-224c6225fe67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:28.258767 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.258762 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-istio-token" pod "ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d" (UID: "ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:24:28.259337 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.259281 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-istio-envoy" pod "ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d" (UID: "ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:28.259682 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.259656 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67b3447-b8b2-4485-bacf-0d39a1352feb-istio-envoy" pod "a67b3447-b8b2-4485-bacf-0d39a1352feb" (UID: "a67b3447-b8b2-4485-bacf-0d39a1352feb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:28.259778 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.259724 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a67b3447-b8b2-4485-bacf-0d39a1352feb-istio-podinfo" pod "a67b3447-b8b2-4485-bacf-0d39a1352feb" (UID: "a67b3447-b8b2-4485-bacf-0d39a1352feb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:24:28.260034 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.260017 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b820762-bd67-4d19-a050-224c6225fe67-istio-token" pod "9b820762-bd67-4d19-a050-224c6225fe67" (UID: "9b820762-bd67-4d19-a050-224c6225fe67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:24:28.260544 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.260517 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b820762-bd67-4d19-a050-224c6225fe67-istio-podinfo" pod "9b820762-bd67-4d19-a050-224c6225fe67" (UID: "9b820762-bd67-4d19-a050-224c6225fe67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:24:28.260642 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.260626 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b820762-bd67-4d19-a050-224c6225fe67-istio-envoy" pod "9b820762-bd67-4d19-a050-224c6225fe67" (UID: "9b820762-bd67-4d19-a050-224c6225fe67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:28.260971 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.260951 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a67b3447-b8b2-4485-bacf-0d39a1352feb-kube-api-access-n8wxz" pod "a67b3447-b8b2-4485-bacf-0d39a1352feb" (UID: "a67b3447-b8b2-4485-bacf-0d39a1352feb"). InnerVolumeSpecName "kube-api-access-n8wxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:24:28.358622 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.358523 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:28.358622 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.358553 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:28.358622 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.358570 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8mr7\" (UniqueName: \"kubernetes.io/projected/9b820762-bd67-4d19-a050-224c6225fe67-kube-api-access-s8mr7\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:28.358622 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.358583 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a67b3447-b8b2-4485-bacf-0d39a1352feb-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:28.358622 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.358596 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg9g2\" (UniqueName: \"kubernetes.io/projected/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-kube-api-access-dg9g2\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:28.358622 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.358610 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67b3447-b8b2-4485-bacf-0d39a1352feb-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:28.358622 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.358625 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:28.359026 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.358638 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b820762-bd67-4d19-a050-224c6225fe67-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:28.359026 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.358649 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b820762-bd67-4d19-a050-224c6225fe67-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:28.359026 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.358662 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8wxz\" (UniqueName: \"kubernetes.io/projected/a67b3447-b8b2-4485-bacf-0d39a1352feb-kube-api-access-n8wxz\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:28.359026 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.358674 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b820762-bd67-4d19-a050-224c6225fe67-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:28.359026 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.358687 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b820762-bd67-4d19-a050-224c6225fe67-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:28.359026 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.358702 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b820762-bd67-4d19-a050-224c6225fe67-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:28.359026 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.358714 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67b3447-b8b2-4485-bacf-0d39a1352feb-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:28.359026 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.358726 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a67b3447-b8b2-4485-bacf-0d39a1352feb-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:28.359026 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.358739 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a67b3447-b8b2-4485-bacf-0d39a1352feb-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:28.359026 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.358751 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a67b3447-b8b2-4485-bacf-0d39a1352feb-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:28.359026 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:28.358763 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:29.095012 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:29.094920 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" event={"ID":"9b820762-bd67-4d19-a050-224c6225fe67","Type":"ContainerDied","Data":"760a2669742b64f85a026c9e62ecc018e0ac5e72ec49524684067b2fbdc507d5"} Nov 02 23:24:29.095012 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:29.094941 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z" Nov 02 23:24:29.095012 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:29.094998 5076 scope.go:122] "RemoveContainer" containerID="7723c5dbb3f60e1d039437c32bc35fdf00bc8c48b4868e4db88417196782c66a" Nov 02 23:24:29.096571 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:29.096543 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" Nov 02 23:24:29.096571 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:29.096551 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8" event={"ID":"ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d","Type":"ContainerDied","Data":"80c2beead924f4047ba6c1ec29fd0aa93946be3999c044f7d210ceee15d6f348"} Nov 02 23:24:29.098124 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:29.098107 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" Nov 02 23:24:29.098501 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:29.098152 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h" event={"ID":"db841b5b-fade-48e4-9515-ef923b888f42","Type":"ContainerDied","Data":"2b0c984aaaff34667ef6334ae47576303eda8c9ba7ebaa2b1cf6375bbe5215b0"} Nov 02 23:24:29.099778 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:29.099758 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" event={"ID":"a67b3447-b8b2-4485-bacf-0d39a1352feb","Type":"ContainerDied","Data":"a21c1fa594f6f1311a629027155e98523c521abfa96b5170f4c1b12f7fca9afa"} Nov 02 23:24:29.099885 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:29.099790 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn" Nov 02 23:24:29.107104 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:29.106764 5076 scope.go:122] "RemoveContainer" containerID="271338a4e786084b47c46ed1176d93ad1ca308b53f49763c1bb02adac63ab9d7" Nov 02 23:24:29.118038 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:29.118023 5076 scope.go:122] "RemoveContainer" containerID="50bebcfe439db99de3dd447638369c5a25e6b60507a29e399e952c04c69fadc9" Nov 02 23:24:29.128028 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:29.128004 5076 scope.go:122] "RemoveContainer" containerID="6a8adf24919c12f4141a053259df0816b40eebbc8bfb7bdea94a8646b5711b41" Nov 02 23:24:29.253344 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:29.253292 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8"] Nov 02 23:24:29.300318 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:29.300282 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v7jp8"] Nov 02 23:24:29.404281 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:29.404201 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z"] Nov 02 23:24:29.411058 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:29.411024 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sc89z"] Nov 02 23:24:29.514851 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:29.514818 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn"] Nov 02 23:24:29.520286 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:29.520260 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ccjjn"] Nov 02 23:24:29.624018 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:29.623985 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h"] Nov 02 23:24:29.629338 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:29.629309 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vfl6h"] Nov 02 23:24:31.151592 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:31.151557 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b820762-bd67-4d19-a050-224c6225fe67" path="/var/lib/kubelet/pods/9b820762-bd67-4d19-a050-224c6225fe67/volumes" Nov 02 23:24:31.152336 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:31.152316 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a67b3447-b8b2-4485-bacf-0d39a1352feb" path="/var/lib/kubelet/pods/a67b3447-b8b2-4485-bacf-0d39a1352feb/volumes" Nov 02 23:24:31.153018 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:31.153001 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d" path="/var/lib/kubelet/pods/ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d/volumes" Nov 02 23:24:31.153857 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:31.153840 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db841b5b-fade-48e4-9515-ef923b888f42" path="/var/lib/kubelet/pods/db841b5b-fade-48e4-9515-ef923b888f42/volumes" Nov 02 23:24:31.760808 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:31.760775 5076 scope.go:122] "RemoveContainer" containerID="323da69353697cbd4bc50143494f278662d9ef246af24a61c917d0e12f21411e" Nov 02 23:24:39.884803 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:24:39.884763 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:24:45.764010 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:45.763928 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-027689-5864dbbf67-j588d"] Nov 02 23:24:45.764598 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:45.764135 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-027689-5864dbbf67-j588d" podUID="105ffb43-f715-44c3-815f-452b3086133f" containerName="maas-api" containerID="cri-o://fc22217d4c450fe836112a0fd2924d35a434af8a9653aa1941c2629715050ca1" gracePeriod=30 Nov 02 23:24:46.142753 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:46.142728 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-027689-5864dbbf67-j588d" Nov 02 23:24:46.239166 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:46.239127 5076 generic.go:356] "Generic (PLEG): container finished" podID="105ffb43-f715-44c3-815f-452b3086133f" containerID="fc22217d4c450fe836112a0fd2924d35a434af8a9653aa1941c2629715050ca1" exitCode=0 Nov 02 23:24:46.239650 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:46.239621 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-027689-5864dbbf67-j588d" event={"ID":"105ffb43-f715-44c3-815f-452b3086133f","Type":"ContainerDied","Data":"fc22217d4c450fe836112a0fd2924d35a434af8a9653aa1941c2629715050ca1"} Nov 02 23:24:46.239839 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:46.239817 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-027689-5864dbbf67-j588d" event={"ID":"105ffb43-f715-44c3-815f-452b3086133f","Type":"ContainerDied","Data":"f94569f4a875a82174f8777471b3d6313ec4e11de226604b0bd58392444f076e"} Nov 02 23:24:46.240238 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:46.240128 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-027689-5864dbbf67-j588d" Nov 02 23:24:46.240519 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:46.240144 5076 scope.go:122] "RemoveContainer" containerID="fc22217d4c450fe836112a0fd2924d35a434af8a9653aa1941c2629715050ca1" Nov 02 23:24:46.264173 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:46.264078 5076 scope.go:122] "RemoveContainer" containerID="fc22217d4c450fe836112a0fd2924d35a434af8a9653aa1941c2629715050ca1" Nov 02 23:24:46.264431 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:24:46.264410 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc22217d4c450fe836112a0fd2924d35a434af8a9653aa1941c2629715050ca1\": container with ID starting with fc22217d4c450fe836112a0fd2924d35a434af8a9653aa1941c2629715050ca1 not found: ID does not exist" containerID="fc22217d4c450fe836112a0fd2924d35a434af8a9653aa1941c2629715050ca1" Nov 02 23:24:46.264523 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:46.264440 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc22217d4c450fe836112a0fd2924d35a434af8a9653aa1941c2629715050ca1"} err="failed to get container status \"fc22217d4c450fe836112a0fd2924d35a434af8a9653aa1941c2629715050ca1\": rpc error: code = NotFound desc = could not find container \"fc22217d4c450fe836112a0fd2924d35a434af8a9653aa1941c2629715050ca1\": container with ID starting with fc22217d4c450fe836112a0fd2924d35a434af8a9653aa1941c2629715050ca1 not found: ID does not exist" Nov 02 23:24:46.296702 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:46.296663 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/105ffb43-f715-44c3-815f-452b3086133f-kube-api-access-zrs8w\" (UniqueName: \"kubernetes.io/projected/105ffb43-f715-44c3-815f-452b3086133f-kube-api-access-zrs8w\") pod \"105ffb43-f715-44c3-815f-452b3086133f\" (UID: \"105ffb43-f715-44c3-815f-452b3086133f\") " Nov 02 23:24:46.296845 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:46.296800 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/105ffb43-f715-44c3-815f-452b3086133f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/105ffb43-f715-44c3-815f-452b3086133f-openshift-service-ca\") pod \"105ffb43-f715-44c3-815f-452b3086133f\" (UID: \"105ffb43-f715-44c3-815f-452b3086133f\") " Nov 02 23:24:46.296978 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:46.296958 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/105ffb43-f715-44c3-815f-452b3086133f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/105ffb43-f715-44c3-815f-452b3086133f-metrics-certs\") pod \"105ffb43-f715-44c3-815f-452b3086133f\" (UID: \"105ffb43-f715-44c3-815f-452b3086133f\") " Nov 02 23:24:46.297103 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:46.297083 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/105ffb43-f715-44c3-815f-452b3086133f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/105ffb43-f715-44c3-815f-452b3086133f-maas-api-tls\") pod \"105ffb43-f715-44c3-815f-452b3086133f\" (UID: \"105ffb43-f715-44c3-815f-452b3086133f\") " Nov 02 23:24:46.297197 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:46.297165 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/105ffb43-f715-44c3-815f-452b3086133f-openshift-service-ca" pod "105ffb43-f715-44c3-815f-452b3086133f" (UID: "105ffb43-f715-44c3-815f-452b3086133f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:24:46.298637 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:46.298609 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/105ffb43-f715-44c3-815f-452b3086133f-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:46.299442 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:46.299410 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/105ffb43-f715-44c3-815f-452b3086133f-metrics-certs" pod "105ffb43-f715-44c3-815f-452b3086133f" (UID: "105ffb43-f715-44c3-815f-452b3086133f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:24:46.299556 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:46.299451 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/105ffb43-f715-44c3-815f-452b3086133f-kube-api-access-zrs8w" pod "105ffb43-f715-44c3-815f-452b3086133f" (UID: "105ffb43-f715-44c3-815f-452b3086133f"). InnerVolumeSpecName "kube-api-access-zrs8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:24:46.299883 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:46.299861 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/105ffb43-f715-44c3-815f-452b3086133f-maas-api-tls" pod "105ffb43-f715-44c3-815f-452b3086133f" (UID: "105ffb43-f715-44c3-815f-452b3086133f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:24:46.339724 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:46.339685 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-5f96f7cdc9-5nrp5"] Nov 02 23:24:46.339978 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:46.339953 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-5f96f7cdc9-5nrp5" podUID="f12c3e68-ee99-4f7d-aeeb-445a9a13a5d3" containerName="payload-pre-processing" containerID="cri-o://3585040fc23a6ac9543abc73d032b0cd881c0f34080364cb2eed8fb09cf93277" gracePeriod=30 Nov 02 23:24:46.399642 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:46.399609 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/105ffb43-f715-44c3-815f-452b3086133f-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:46.399642 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:46.399640 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/105ffb43-f715-44c3-815f-452b3086133f-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:46.399830 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:46.399651 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrs8w\" (UniqueName: \"kubernetes.io/projected/105ffb43-f715-44c3-815f-452b3086133f-kube-api-access-zrs8w\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:46.645643 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:46.645617 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-5f96f7cdc9-5nrp5" Nov 02 23:24:46.674135 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:46.674098 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-027689-5864dbbf67-j588d"] Nov 02 23:24:46.681114 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:46.681083 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-027689-5864dbbf67-j588d"] Nov 02 23:24:46.830917 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:46.817453 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f12c3e68-ee99-4f7d-aeeb-445a9a13a5d3-kube-api-access-q4xgc\" (UniqueName: \"kubernetes.io/projected/f12c3e68-ee99-4f7d-aeeb-445a9a13a5d3-kube-api-access-q4xgc\") pod \"f12c3e68-ee99-4f7d-aeeb-445a9a13a5d3\" (UID: \"f12c3e68-ee99-4f7d-aeeb-445a9a13a5d3\") " Nov 02 23:24:46.830917 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:46.817573 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f12c3e68-ee99-4f7d-aeeb-445a9a13a5d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f12c3e68-ee99-4f7d-aeeb-445a9a13a5d3-plugins-config-volume\") pod \"f12c3e68-ee99-4f7d-aeeb-445a9a13a5d3\" (UID: \"f12c3e68-ee99-4f7d-aeeb-445a9a13a5d3\") " Nov 02 23:24:46.830917 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:46.818409 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f12c3e68-ee99-4f7d-aeeb-445a9a13a5d3-plugins-config-volume" pod "f12c3e68-ee99-4f7d-aeeb-445a9a13a5d3" (UID: "f12c3e68-ee99-4f7d-aeeb-445a9a13a5d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:24:46.883659 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:46.883061 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f12c3e68-ee99-4f7d-aeeb-445a9a13a5d3-kube-api-access-q4xgc" pod "f12c3e68-ee99-4f7d-aeeb-445a9a13a5d3" (UID: "f12c3e68-ee99-4f7d-aeeb-445a9a13a5d3"). InnerVolumeSpecName "kube-api-access-q4xgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:24:46.966572 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:46.964996 5076 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f12c3e68-ee99-4f7d-aeeb-445a9a13a5d3-plugins-config-volume\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:46.966572 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:46.965038 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4xgc\" (UniqueName: \"kubernetes.io/projected/f12c3e68-ee99-4f7d-aeeb-445a9a13a5d3-kube-api-access-q4xgc\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:47.148034 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:47.148003 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="105ffb43-f715-44c3-815f-452b3086133f" path="/var/lib/kubelet/pods/105ffb43-f715-44c3-815f-452b3086133f/volumes" Nov 02 23:24:47.247880 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:47.247792 5076 generic.go:356] "Generic (PLEG): container finished" podID="f12c3e68-ee99-4f7d-aeeb-445a9a13a5d3" containerID="3585040fc23a6ac9543abc73d032b0cd881c0f34080364cb2eed8fb09cf93277" exitCode=0 Nov 02 23:24:47.247880 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:47.247838 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-5f96f7cdc9-5nrp5" Nov 02 23:24:47.247880 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:47.247845 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-5f96f7cdc9-5nrp5" event={"ID":"f12c3e68-ee99-4f7d-aeeb-445a9a13a5d3","Type":"ContainerDied","Data":"3585040fc23a6ac9543abc73d032b0cd881c0f34080364cb2eed8fb09cf93277"} Nov 02 23:24:47.248140 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:47.247902 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-5f96f7cdc9-5nrp5" event={"ID":"f12c3e68-ee99-4f7d-aeeb-445a9a13a5d3","Type":"ContainerDied","Data":"7c8ded6ffea2afd7860cad9d5a8bd8a9ccc63a549c24a04d7e3372a47b320d67"} Nov 02 23:24:47.248140 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:47.247926 5076 scope.go:122] "RemoveContainer" containerID="3585040fc23a6ac9543abc73d032b0cd881c0f34080364cb2eed8fb09cf93277" Nov 02 23:24:47.257695 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:47.257678 5076 scope.go:122] "RemoveContainer" containerID="3585040fc23a6ac9543abc73d032b0cd881c0f34080364cb2eed8fb09cf93277" Nov 02 23:24:47.257965 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:24:47.257946 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3585040fc23a6ac9543abc73d032b0cd881c0f34080364cb2eed8fb09cf93277\": container with ID starting with 3585040fc23a6ac9543abc73d032b0cd881c0f34080364cb2eed8fb09cf93277 not found: ID does not exist" containerID="3585040fc23a6ac9543abc73d032b0cd881c0f34080364cb2eed8fb09cf93277" Nov 02 23:24:47.258020 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:47.257973 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3585040fc23a6ac9543abc73d032b0cd881c0f34080364cb2eed8fb09cf93277"} err="failed to get container status \"3585040fc23a6ac9543abc73d032b0cd881c0f34080364cb2eed8fb09cf93277\": rpc error: code = NotFound desc = could not find container \"3585040fc23a6ac9543abc73d032b0cd881c0f34080364cb2eed8fb09cf93277\": container with ID starting with 3585040fc23a6ac9543abc73d032b0cd881c0f34080364cb2eed8fb09cf93277 not found: ID does not exist" Nov 02 23:24:47.403883 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:47.403848 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-5f96f7cdc9-5nrp5"] Nov 02 23:24:47.408059 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:47.408030 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-027689-5f96f7cdc9-5nrp5"] Nov 02 23:24:49.148938 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:49.148900 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f12c3e68-ee99-4f7d-aeeb-445a9a13a5d3" path="/var/lib/kubelet/pods/f12c3e68-ee99-4f7d-aeeb-445a9a13a5d3/volumes" Nov 02 23:24:53.290835 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:53.290797 5076 generic.go:356] "Generic (PLEG): container finished" podID="86a56d46-242b-4863-8b05-4c5054fa0d90" containerID="e636865b78586e81385059a5a941351a8e90000c6b818967049bdcd15cceb128" exitCode=0 Nov 02 23:24:53.291293 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:53.290867 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" event={"ID":"86a56d46-242b-4863-8b05-4c5054fa0d90","Type":"ContainerDied","Data":"e636865b78586e81385059a5a941351a8e90000c6b818967049bdcd15cceb128"} Nov 02 23:24:53.291293 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:53.291287 5076 scope.go:122] "RemoveContainer" containerID="e636865b78586e81385059a5a941351a8e90000c6b818967049bdcd15cceb128" Nov 02 23:24:53.292854 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:53.292829 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:24:53.292952 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:53.292910 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:24:53.292952 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:53.292948 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:24:54.300658 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:54.300613 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" event={"ID":"86a56d46-242b-4863-8b05-4c5054fa0d90","Type":"ContainerStarted","Data":"6d52a3e9f0fbb68c9c4f419d3e730d2b9da8929c1e564d53d016710d0ae6b590"} Nov 02 23:24:54.800768 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:54.800736 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" Nov 02 23:24:54.800950 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:54.800878 5076 patch_prober.go:29] interesting pod/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 02 23:24:54.800950 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:54.800941 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" podUID="86a56d46-242b-4863-8b05-4c5054fa0d90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 02 23:24:55.800975 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:55.800941 5076 patch_prober.go:29] interesting pod/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 02 23:24:55.801370 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:55.800993 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" podUID="86a56d46-242b-4863-8b05-4c5054fa0d90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 02 23:24:56.543219 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:56.543175 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62"] Nov 02 23:24:56.543437 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:56.543400 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" podUID="801851c2-20cb-4816-9a90-032a209b4933" containerName="istio-proxy" containerID="cri-o://e682adfcca449a0b35361d4d070d3ab6e1120b12afa8e688c98d0a99dab77b1e" gracePeriod=30 Nov 02 23:24:56.583876 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:56.583835 5076 patch_prober.go:29] interesting pod/e2e-ipp-a-027689-openshift-default-749945876c-n6p62 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:24:56.584067 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:56.583909 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" podUID="801851c2-20cb-4816-9a90-032a209b4933" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:24:56.800788 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:56.800678 5076 patch_prober.go:29] interesting pod/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 02 23:24:56.800788 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:56.800769 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" podUID="86a56d46-242b-4863-8b05-4c5054fa0d90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 02 23:24:57.804944 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:57.800667 5076 patch_prober.go:29] interesting pod/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 02 23:24:57.804944 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:57.800728 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" podUID="86a56d46-242b-4863-8b05-4c5054fa0d90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 02 23:24:58.633117 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:58.633062 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-78b54bb9f9-rtscx"] Nov 02 23:24:58.634166 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:58.634132 5076 cpu_manager.go:414] "containerMap: removing container" podUID="9b820762-bd67-4d19-a050-224c6225fe67" containerName="istio-proxy" Nov 02 23:24:58.634166 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:58.634161 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b820762-bd67-4d19-a050-224c6225fe67" containerName="istio-proxy" Nov 02 23:24:58.634361 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:58.634178 5076 cpu_manager.go:414] "containerMap: removing container" podUID="105ffb43-f715-44c3-815f-452b3086133f" containerName="maas-api" Nov 02 23:24:58.634361 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:58.634186 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="105ffb43-f715-44c3-815f-452b3086133f" containerName="maas-api" Nov 02 23:24:58.634361 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:58.634205 5076 cpu_manager.go:414] "containerMap: removing container" podUID="f12c3e68-ee99-4f7d-aeeb-445a9a13a5d3" containerName="payload-pre-processing" Nov 02 23:24:58.634361 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:58.634212 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f12c3e68-ee99-4f7d-aeeb-445a9a13a5d3" containerName="payload-pre-processing" Nov 02 23:24:58.634361 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:58.634246 5076 cpu_manager.go:414] "containerMap: removing container" podUID="ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d" containerName="istio-proxy" Nov 02 23:24:58.634361 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:58.634253 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d" containerName="istio-proxy" Nov 02 23:24:58.634361 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:58.634265 5076 cpu_manager.go:414] "containerMap: removing container" podUID="db841b5b-fade-48e4-9515-ef923b888f42" containerName="istio-proxy" Nov 02 23:24:58.634361 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:58.634272 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db841b5b-fade-48e4-9515-ef923b888f42" containerName="istio-proxy" Nov 02 23:24:58.634361 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:58.634295 5076 cpu_manager.go:414] "containerMap: removing container" podUID="a67b3447-b8b2-4485-bacf-0d39a1352feb" containerName="istio-proxy" Nov 02 23:24:58.634361 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:58.634302 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a67b3447-b8b2-4485-bacf-0d39a1352feb" containerName="istio-proxy" Nov 02 23:24:58.634361 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:58.634315 5076 cpu_manager.go:414] "containerMap: removing container" podUID="5b6815fb-2006-4710-a166-db066f8359f8" containerName="istio-proxy" Nov 02 23:24:58.634361 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:58.634322 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b6815fb-2006-4710-a166-db066f8359f8" containerName="istio-proxy" Nov 02 23:24:58.634978 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:58.634494 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="a67b3447-b8b2-4485-bacf-0d39a1352feb" containerName="istio-proxy" Nov 02 23:24:58.634978 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:58.634508 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b6815fb-2006-4710-a166-db066f8359f8" containerName="istio-proxy" Nov 02 23:24:58.634978 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:58.634526 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b820762-bd67-4d19-a050-224c6225fe67" containerName="istio-proxy" Nov 02 23:24:58.634978 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:58.634537 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="105ffb43-f715-44c3-815f-452b3086133f" containerName="maas-api" Nov 02 23:24:58.634978 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:58.634547 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="f12c3e68-ee99-4f7d-aeeb-445a9a13a5d3" containerName="payload-pre-processing" Nov 02 23:24:58.634978 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:58.634560 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad898fd0-2ba1-4e99-ae5a-14dd3afb3d8d" containerName="istio-proxy" Nov 02 23:24:58.634978 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:58.634568 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="db841b5b-fade-48e4-9515-ef923b888f42" containerName="istio-proxy" Nov 02 23:24:58.638628 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:58.638604 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-78b54bb9f9-rtscx" Nov 02 23:24:58.685762 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:58.685726 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-78b54bb9f9-rtscx"] Nov 02 23:24:58.732231 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:58.732195 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6a894c\"" Nov 02 23:24:58.733137 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:58.733107 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-6a894c-dockercfg-gzkg6\"" Nov 02 23:24:58.741054 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:58.741019 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bn9s\" (UniqueName: \"kubernetes.io/projected/0f98b219-1d3d-47d4-aaf1-3d06149307eb-kube-api-access-5bn9s\") pod \"payload-pre-processing-e2e-ipp-a-6a894c-78b54bb9f9-rtscx\" (UID: \"0f98b219-1d3d-47d4-aaf1-3d06149307eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-78b54bb9f9-rtscx" Nov 02 23:24:58.741263 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:58.741240 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f98b219-1d3d-47d4-aaf1-3d06149307eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6a894c-78b54bb9f9-rtscx\" (UID: \"0f98b219-1d3d-47d4-aaf1-3d06149307eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-78b54bb9f9-rtscx" Nov 02 23:24:58.800436 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:58.800405 5076 patch_prober.go:29] interesting pod/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 02 23:24:58.800605 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:58.800451 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" podUID="86a56d46-242b-4863-8b05-4c5054fa0d90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 02 23:24:58.842113 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:58.842068 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bn9s\" (UniqueName: \"kubernetes.io/projected/0f98b219-1d3d-47d4-aaf1-3d06149307eb-kube-api-access-5bn9s\") pod \"payload-pre-processing-e2e-ipp-a-6a894c-78b54bb9f9-rtscx\" (UID: \"0f98b219-1d3d-47d4-aaf1-3d06149307eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-78b54bb9f9-rtscx" Nov 02 23:24:58.842591 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:58.842228 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f98b219-1d3d-47d4-aaf1-3d06149307eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6a894c-78b54bb9f9-rtscx\" (UID: \"0f98b219-1d3d-47d4-aaf1-3d06149307eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-78b54bb9f9-rtscx" Nov 02 23:24:58.843061 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:58.843035 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f98b219-1d3d-47d4-aaf1-3d06149307eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6a894c-78b54bb9f9-rtscx\" (UID: \"0f98b219-1d3d-47d4-aaf1-3d06149307eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-78b54bb9f9-rtscx" Nov 02 23:24:58.902780 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:58.902708 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bn9s\" (UniqueName: \"kubernetes.io/projected/0f98b219-1d3d-47d4-aaf1-3d06149307eb-kube-api-access-5bn9s\") pod \"payload-pre-processing-e2e-ipp-a-6a894c-78b54bb9f9-rtscx\" (UID: \"0f98b219-1d3d-47d4-aaf1-3d06149307eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-78b54bb9f9-rtscx" Nov 02 23:24:58.954848 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:58.954771 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-78b54bb9f9-rtscx" Nov 02 23:24:59.035300 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.032743 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4"] Nov 02 23:24:59.073933 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.073874 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" Nov 02 23:24:59.079356 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.078420 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4"] Nov 02 23:24:59.138915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.125848 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-443fdf-openshift-default-dockercfg-5pv7j\"" Nov 02 23:24:59.197319 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.196951 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-istiod-ca-cert\") pod \"e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4\" (UID: \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\") " pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" Nov 02 23:24:59.197319 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.197019 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-workload-socket\") pod \"e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4\" (UID: \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\") " pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" Nov 02 23:24:59.197319 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.197095 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-istio-podinfo\") pod \"e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4\" (UID: \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\") " pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" Nov 02 23:24:59.198038 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.198006 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-credential-socket\") pod \"e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4\" (UID: \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\") " pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" Nov 02 23:24:59.198107 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.198064 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6xpv\" (UniqueName: \"kubernetes.io/projected/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-kube-api-access-w6xpv\") pod \"e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4\" (UID: \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\") " pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" Nov 02 23:24:59.198144 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.198105 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-istio-token\") pod \"e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4\" (UID: \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\") " pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" Nov 02 23:24:59.198210 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.198193 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-istio-data\") pod \"e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4\" (UID: \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\") " pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" Nov 02 23:24:59.198272 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.198257 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-workload-certs\") pod \"e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4\" (UID: \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\") " pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" Nov 02 23:24:59.198376 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.198325 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-istio-envoy\") pod \"e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4\" (UID: \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\") " pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" Nov 02 23:24:59.300033 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.300002 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-workload-socket\") pod \"e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4\" (UID: \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\") " pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" Nov 02 23:24:59.300224 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.300078 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-istio-podinfo\") pod \"e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4\" (UID: \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\") " pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" Nov 02 23:24:59.300224 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.300144 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-credential-socket\") pod \"e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4\" (UID: \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\") " pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" Nov 02 23:24:59.300224 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.300180 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6xpv\" (UniqueName: \"kubernetes.io/projected/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-kube-api-access-w6xpv\") pod \"e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4\" (UID: \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\") " pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" Nov 02 23:24:59.300224 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.300212 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-istio-token\") pod \"e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4\" (UID: \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\") " pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" Nov 02 23:24:59.300468 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.300314 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-istio-data\") pod \"e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4\" (UID: \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\") " pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" Nov 02 23:24:59.300468 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.300368 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-workload-certs\") pod \"e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4\" (UID: \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\") " pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" Nov 02 23:24:59.300468 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.300403 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-istio-envoy\") pod \"e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4\" (UID: \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\") " pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" Nov 02 23:24:59.300468 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.300461 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-istiod-ca-cert\") pod \"e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4\" (UID: \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\") " pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" Nov 02 23:24:59.300729 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.300494 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-workload-socket\") pod \"e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4\" (UID: \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\") " pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" Nov 02 23:24:59.300867 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.300764 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-credential-socket\") pod \"e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4\" (UID: \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\") " pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" Nov 02 23:24:59.301058 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.301034 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-workload-certs\") pod \"e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4\" (UID: \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\") " pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" Nov 02 23:24:59.301199 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.301163 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-istio-data\") pod \"e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4\" (UID: \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\") " pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" Nov 02 23:24:59.301457 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.301432 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-istiod-ca-cert\") pod \"e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4\" (UID: \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\") " pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" Nov 02 23:24:59.303152 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.303128 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-istio-envoy\") pod \"e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4\" (UID: \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\") " pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" Nov 02 23:24:59.303629 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.303608 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-istio-podinfo\") pod \"e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4\" (UID: \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\") " pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" Nov 02 23:24:59.355023 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.354990 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6xpv\" (UniqueName: \"kubernetes.io/projected/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-kube-api-access-w6xpv\") pod \"e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4\" (UID: \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\") " pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" Nov 02 23:24:59.355174 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.355096 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-istio-token\") pod \"e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4\" (UID: \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\") " pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" Nov 02 23:24:59.367116 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.367085 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6a894c-5dbb4748-58dmg"] Nov 02 23:24:59.372252 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.372233 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a894c-5dbb4748-58dmg" Nov 02 23:24:59.394171 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.394147 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" Nov 02 23:24:59.440765 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.440731 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6a894c-5dbb4748-58dmg"] Nov 02 23:24:59.498679 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.497253 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-78b54bb9f9-rtscx"] Nov 02 23:24:59.504622 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.504564 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nhlp\" (UniqueName: \"kubernetes.io/projected/7542df71-b433-4496-b175-1797e098f9fe-kube-api-access-8nhlp\") pod \"payload-processing-e2e-ipp-a-6a894c-5dbb4748-58dmg\" (UID: \"7542df71-b433-4496-b175-1797e098f9fe\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6a894c-5dbb4748-58dmg" Nov 02 23:24:59.505003 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.504977 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7542df71-b433-4496-b175-1797e098f9fe-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6a894c-5dbb4748-58dmg\" (UID: \"7542df71-b433-4496-b175-1797e098f9fe\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6a894c-5dbb4748-58dmg" Nov 02 23:24:59.606465 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.606440 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nhlp\" (UniqueName: \"kubernetes.io/projected/7542df71-b433-4496-b175-1797e098f9fe-kube-api-access-8nhlp\") pod \"payload-processing-e2e-ipp-a-6a894c-5dbb4748-58dmg\" (UID: \"7542df71-b433-4496-b175-1797e098f9fe\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6a894c-5dbb4748-58dmg" Nov 02 23:24:59.606582 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.606488 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7542df71-b433-4496-b175-1797e098f9fe-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6a894c-5dbb4748-58dmg\" (UID: \"7542df71-b433-4496-b175-1797e098f9fe\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6a894c-5dbb4748-58dmg" Nov 02 23:24:59.607191 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.607173 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7542df71-b433-4496-b175-1797e098f9fe-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6a894c-5dbb4748-58dmg\" (UID: \"7542df71-b433-4496-b175-1797e098f9fe\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6a894c-5dbb4748-58dmg" Nov 02 23:24:59.660875 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.660835 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nhlp\" (UniqueName: \"kubernetes.io/projected/7542df71-b433-4496-b175-1797e098f9fe-kube-api-access-8nhlp\") pod \"payload-processing-e2e-ipp-a-6a894c-5dbb4748-58dmg\" (UID: \"7542df71-b433-4496-b175-1797e098f9fe\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6a894c-5dbb4748-58dmg" Nov 02 23:24:59.686561 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.686535 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a894c-5dbb4748-58dmg" Nov 02 23:24:59.800702 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.800667 5076 patch_prober.go:29] interesting pod/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 02 23:24:59.800870 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.800730 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" podUID="86a56d46-242b-4863-8b05-4c5054fa0d90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 02 23:24:59.809854 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:24:59.809815 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c5807a1_409d_4188_abb7_1bf3bfb8e07d.slice/crio-b3101d3965b07a98b73d74f47e6426e6e4814118b9a885c984c3566eb5bee42e WatchSource:0}: Error finding container b3101d3965b07a98b73d74f47e6426e6e4814118b9a885c984c3566eb5bee42e: Status 404 returned error can't find the container with id b3101d3965b07a98b73d74f47e6426e6e4814118b9a885c984c3566eb5bee42e Nov 02 23:24:59.811850 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.811822 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4"] Nov 02 23:24:59.813098 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.813061 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:24:59.813161 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.813148 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:24:59.813207 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:24:59.813192 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:25:00.058513 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:25:00.058478 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7542df71_b433_4496_b175_1797e098f9fe.slice/crio-d820e326da4599e69036ed9a3223d610ae7137c9771079a168f59bef0b6dd8c8 WatchSource:0}: Error finding container d820e326da4599e69036ed9a3223d610ae7137c9771079a168f59bef0b6dd8c8: Status 404 returned error can't find the container with id d820e326da4599e69036ed9a3223d610ae7137c9771079a168f59bef0b6dd8c8 Nov 02 23:25:00.059349 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:00.059317 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6a894c-5dbb4748-58dmg"] Nov 02 23:25:00.359731 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:00.359625 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a894c-5dbb4748-58dmg" event={"ID":"7542df71-b433-4496-b175-1797e098f9fe","Type":"ContainerStarted","Data":"78d74cc7cfd646c8520d9a00f89449f4b8ec436e6ede934dba43ce7271a2fa14"} Nov 02 23:25:00.359731 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:00.359690 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a894c-5dbb4748-58dmg" event={"ID":"7542df71-b433-4496-b175-1797e098f9fe","Type":"ContainerStarted","Data":"d820e326da4599e69036ed9a3223d610ae7137c9771079a168f59bef0b6dd8c8"} Nov 02 23:25:00.359974 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:00.359810 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a894c-5dbb4748-58dmg" Nov 02 23:25:00.361351 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:00.361325 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-78b54bb9f9-rtscx" event={"ID":"0f98b219-1d3d-47d4-aaf1-3d06149307eb","Type":"ContainerStarted","Data":"92ce1fa84d3d751e6d06cd2640b0a5c19cfe359c5dc5755b0c6be09fa9e76f19"} Nov 02 23:25:00.361471 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:00.361371 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-78b54bb9f9-rtscx" event={"ID":"0f98b219-1d3d-47d4-aaf1-3d06149307eb","Type":"ContainerStarted","Data":"716f10a1f14379b39e13f1edc8f6b6094531c5c27a118f3266d61d499b8fa3f7"} Nov 02 23:25:00.361471 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:00.361398 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-78b54bb9f9-rtscx" Nov 02 23:25:00.363057 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:00.363039 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" event={"ID":"8c5807a1-409d-4188-abb7-1bf3bfb8e07d","Type":"ContainerStarted","Data":"bb3b65d54e0566441a9ab3fadbc4e9cd77ea6305086388174b1fd87a19c4fdf6"} Nov 02 23:25:00.363134 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:00.363072 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" event={"ID":"8c5807a1-409d-4188-abb7-1bf3bfb8e07d","Type":"ContainerStarted","Data":"b3101d3965b07a98b73d74f47e6426e6e4814118b9a885c984c3566eb5bee42e"} Nov 02 23:25:00.395116 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:00.395087 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" Nov 02 23:25:00.400468 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:00.400446 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" Nov 02 23:25:00.470236 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:00.470156 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a894c-5dbb4748-58dmg" podStartSLOduration=1.470142622 podStartE2EDuration="1.470142622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:24:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:25:00.466924766 +0000 UTC m=+1949.435456547" watchObservedRunningTime="2026-11-02 23:25:00.470142622 +0000 UTC m=+1949.438674405" Nov 02 23:25:00.735919 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:00.732024 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-78b54bb9f9-rtscx" podStartSLOduration=2.731999394 podStartE2EDuration="2.731999394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:24:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:25:00.715332566 +0000 UTC m=+1949.683864350" watchObservedRunningTime="2026-11-02 23:25:00.731999394 +0000 UTC m=+1949.700531179" Nov 02 23:25:00.735919 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:00.732452 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" podStartSLOduration=2.732436795 podStartE2EDuration="2.732436795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:24:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:25:00.60758912 +0000 UTC m=+1949.576120933" watchObservedRunningTime="2026-11-02 23:25:00.732436795 +0000 UTC m=+1949.700968611" Nov 02 23:25:00.805817 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:00.805782 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" Nov 02 23:25:00.806239 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:00.806217 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" Nov 02 23:25:00.806812 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:00.806790 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" Nov 02 23:25:01.370696 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:01.370665 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" Nov 02 23:25:01.371678 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:01.371657 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" Nov 02 23:25:01.835231 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:01.835204 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" Nov 02 23:25:01.881462 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:01.881433 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/801851c2-20cb-4816-9a90-032a209b4933-kube-api-access-7rvfc\" (UniqueName: \"kubernetes.io/projected/801851c2-20cb-4816-9a90-032a209b4933-kube-api-access-7rvfc\") pod \"801851c2-20cb-4816-9a90-032a209b4933\" (UID: \"801851c2-20cb-4816-9a90-032a209b4933\") " Nov 02 23:25:01.881632 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:01.881477 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/801851c2-20cb-4816-9a90-032a209b4933-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801851c2-20cb-4816-9a90-032a209b4933-workload-certs\") pod \"801851c2-20cb-4816-9a90-032a209b4933\" (UID: \"801851c2-20cb-4816-9a90-032a209b4933\") " Nov 02 23:25:01.881632 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:01.881531 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/801851c2-20cb-4816-9a90-032a209b4933-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801851c2-20cb-4816-9a90-032a209b4933-istio-podinfo\") pod \"801851c2-20cb-4816-9a90-032a209b4933\" (UID: \"801851c2-20cb-4816-9a90-032a209b4933\") " Nov 02 23:25:01.881632 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:01.881553 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/801851c2-20cb-4816-9a90-032a209b4933-istio-token\" (UniqueName: \"kubernetes.io/projected/801851c2-20cb-4816-9a90-032a209b4933-istio-token\") pod \"801851c2-20cb-4816-9a90-032a209b4933\" (UID: \"801851c2-20cb-4816-9a90-032a209b4933\") " Nov 02 23:25:01.881632 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:01.881578 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/801851c2-20cb-4816-9a90-032a209b4933-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801851c2-20cb-4816-9a90-032a209b4933-workload-socket\") pod \"801851c2-20cb-4816-9a90-032a209b4933\" (UID: \"801851c2-20cb-4816-9a90-032a209b4933\") " Nov 02 23:25:01.882003 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:01.881694 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/801851c2-20cb-4816-9a90-032a209b4933-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801851c2-20cb-4816-9a90-032a209b4933-credential-socket\") pod \"801851c2-20cb-4816-9a90-032a209b4933\" (UID: \"801851c2-20cb-4816-9a90-032a209b4933\") " Nov 02 23:25:01.882118 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:01.881968 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801851c2-20cb-4816-9a90-032a209b4933-credential-socket" pod "801851c2-20cb-4816-9a90-032a209b4933" (UID: "801851c2-20cb-4816-9a90-032a209b4933"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:01.882184 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:01.882074 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801851c2-20cb-4816-9a90-032a209b4933-workload-socket" pod "801851c2-20cb-4816-9a90-032a209b4933" (UID: "801851c2-20cb-4816-9a90-032a209b4933"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:01.882184 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:01.882087 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801851c2-20cb-4816-9a90-032a209b4933-workload-certs" pod "801851c2-20cb-4816-9a90-032a209b4933" (UID: "801851c2-20cb-4816-9a90-032a209b4933"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:01.882277 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:01.882198 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/801851c2-20cb-4816-9a90-032a209b4933-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801851c2-20cb-4816-9a90-032a209b4933-istiod-ca-cert\") pod \"801851c2-20cb-4816-9a90-032a209b4933\" (UID: \"801851c2-20cb-4816-9a90-032a209b4933\") " Nov 02 23:25:01.882330 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:01.882312 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/801851c2-20cb-4816-9a90-032a209b4933-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801851c2-20cb-4816-9a90-032a209b4933-istio-data\") pod \"801851c2-20cb-4816-9a90-032a209b4933\" (UID: \"801851c2-20cb-4816-9a90-032a209b4933\") " Nov 02 23:25:01.882467 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:01.882449 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/801851c2-20cb-4816-9a90-032a209b4933-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801851c2-20cb-4816-9a90-032a209b4933-istio-envoy\") pod \"801851c2-20cb-4816-9a90-032a209b4933\" (UID: \"801851c2-20cb-4816-9a90-032a209b4933\") " Nov 02 23:25:01.882593 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:01.882572 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801851c2-20cb-4816-9a90-032a209b4933-istio-data" pod "801851c2-20cb-4816-9a90-032a209b4933" (UID: "801851c2-20cb-4816-9a90-032a209b4933"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:01.882647 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:01.882607 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/801851c2-20cb-4816-9a90-032a209b4933-istiod-ca-cert" pod "801851c2-20cb-4816-9a90-032a209b4933" (UID: "801851c2-20cb-4816-9a90-032a209b4933"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:25:01.883314 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:01.883287 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801851c2-20cb-4816-9a90-032a209b4933-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:01.883314 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:01.883317 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801851c2-20cb-4816-9a90-032a209b4933-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:01.883512 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:01.883332 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801851c2-20cb-4816-9a90-032a209b4933-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:01.883512 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:01.883346 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801851c2-20cb-4816-9a90-032a209b4933-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:01.883512 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:01.883362 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801851c2-20cb-4816-9a90-032a209b4933-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:01.884263 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:01.884231 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/801851c2-20cb-4816-9a90-032a209b4933-istio-podinfo" pod "801851c2-20cb-4816-9a90-032a209b4933" (UID: "801851c2-20cb-4816-9a90-032a209b4933"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:25:01.884381 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:01.884294 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/801851c2-20cb-4816-9a90-032a209b4933-kube-api-access-7rvfc" pod "801851c2-20cb-4816-9a90-032a209b4933" (UID: "801851c2-20cb-4816-9a90-032a209b4933"). InnerVolumeSpecName "kube-api-access-7rvfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:25:01.884576 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:01.884551 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/801851c2-20cb-4816-9a90-032a209b4933-istio-token" pod "801851c2-20cb-4816-9a90-032a209b4933" (UID: "801851c2-20cb-4816-9a90-032a209b4933"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:25:01.885175 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:01.885151 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801851c2-20cb-4816-9a90-032a209b4933-istio-envoy" pod "801851c2-20cb-4816-9a90-032a209b4933" (UID: "801851c2-20cb-4816-9a90-032a209b4933"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:01.985323 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:01.985282 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801851c2-20cb-4816-9a90-032a209b4933-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:01.985323 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:01.985322 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rvfc\" (UniqueName: \"kubernetes.io/projected/801851c2-20cb-4816-9a90-032a209b4933-kube-api-access-7rvfc\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:01.985590 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:01.985340 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801851c2-20cb-4816-9a90-032a209b4933-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:01.985590 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:01.985355 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/801851c2-20cb-4816-9a90-032a209b4933-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:02.378681 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:02.378592 5076 generic.go:356] "Generic (PLEG): container finished" podID="801851c2-20cb-4816-9a90-032a209b4933" containerID="e682adfcca449a0b35361d4d070d3ab6e1120b12afa8e688c98d0a99dab77b1e" exitCode=0 Nov 02 23:25:02.378681 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:02.378640 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" Nov 02 23:25:02.379167 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:02.378685 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" event={"ID":"801851c2-20cb-4816-9a90-032a209b4933","Type":"ContainerDied","Data":"e682adfcca449a0b35361d4d070d3ab6e1120b12afa8e688c98d0a99dab77b1e"} Nov 02 23:25:02.379167 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:02.378743 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62" event={"ID":"801851c2-20cb-4816-9a90-032a209b4933","Type":"ContainerDied","Data":"b114d5ff15c3ae93edac81f757b51a389f533171c8324225930324c8e705ac87"} Nov 02 23:25:02.379167 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:02.378767 5076 scope.go:122] "RemoveContainer" containerID="e682adfcca449a0b35361d4d070d3ab6e1120b12afa8e688c98d0a99dab77b1e" Nov 02 23:25:02.390007 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:02.389989 5076 scope.go:122] "RemoveContainer" containerID="b9fdc573e3d71a04eddb008d6f94ca045c652354c2f97f4f60d06081b0adf7ce" Nov 02 23:25:02.399617 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:02.399594 5076 scope.go:122] "RemoveContainer" containerID="e682adfcca449a0b35361d4d070d3ab6e1120b12afa8e688c98d0a99dab77b1e" Nov 02 23:25:02.399925 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:25:02.399874 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e682adfcca449a0b35361d4d070d3ab6e1120b12afa8e688c98d0a99dab77b1e\": container with ID starting with e682adfcca449a0b35361d4d070d3ab6e1120b12afa8e688c98d0a99dab77b1e not found: ID does not exist" containerID="e682adfcca449a0b35361d4d070d3ab6e1120b12afa8e688c98d0a99dab77b1e" Nov 02 23:25:02.400028 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:02.399937 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e682adfcca449a0b35361d4d070d3ab6e1120b12afa8e688c98d0a99dab77b1e"} err="failed to get container status \"e682adfcca449a0b35361d4d070d3ab6e1120b12afa8e688c98d0a99dab77b1e\": rpc error: code = NotFound desc = could not find container \"e682adfcca449a0b35361d4d070d3ab6e1120b12afa8e688c98d0a99dab77b1e\": container with ID starting with e682adfcca449a0b35361d4d070d3ab6e1120b12afa8e688c98d0a99dab77b1e not found: ID does not exist" Nov 02 23:25:02.400028 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:02.399961 5076 scope.go:122] "RemoveContainer" containerID="b9fdc573e3d71a04eddb008d6f94ca045c652354c2f97f4f60d06081b0adf7ce" Nov 02 23:25:02.400230 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:25:02.400211 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9fdc573e3d71a04eddb008d6f94ca045c652354c2f97f4f60d06081b0adf7ce\": container with ID starting with b9fdc573e3d71a04eddb008d6f94ca045c652354c2f97f4f60d06081b0adf7ce not found: ID does not exist" containerID="b9fdc573e3d71a04eddb008d6f94ca045c652354c2f97f4f60d06081b0adf7ce" Nov 02 23:25:02.400287 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:02.400243 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9fdc573e3d71a04eddb008d6f94ca045c652354c2f97f4f60d06081b0adf7ce"} err="failed to get container status \"b9fdc573e3d71a04eddb008d6f94ca045c652354c2f97f4f60d06081b0adf7ce\": rpc error: code = NotFound desc = could not find container \"b9fdc573e3d71a04eddb008d6f94ca045c652354c2f97f4f60d06081b0adf7ce\": container with ID starting with b9fdc573e3d71a04eddb008d6f94ca045c652354c2f97f4f60d06081b0adf7ce not found: ID does not exist" Nov 02 23:25:02.535040 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:02.535006 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62"] Nov 02 23:25:02.582771 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:02.582736 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-027689-openshift-default-749945876c-n6p62"] Nov 02 23:25:03.148779 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:03.148744 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="801851c2-20cb-4816-9a90-032a209b4933" path="/var/lib/kubelet/pods/801851c2-20cb-4816-9a90-032a209b4933/volumes" Nov 02 23:25:09.883086 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:25:09.883048 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:25:11.373196 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:11.373162 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-78b54bb9f9-rtscx" Nov 02 23:25:11.373635 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:11.373247 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a894c-5dbb4748-58dmg" Nov 02 23:25:13.845946 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:13.844768 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9"] Nov 02 23:25:13.845946 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:13.845794 5076 cpu_manager.go:414] "containerMap: removing container" podUID="801851c2-20cb-4816-9a90-032a209b4933" containerName="istio-proxy" Nov 02 23:25:13.845946 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:13.845816 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="801851c2-20cb-4816-9a90-032a209b4933" containerName="istio-proxy" Nov 02 23:25:13.845946 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:13.845843 5076 cpu_manager.go:414] "containerMap: removing container" podUID="801851c2-20cb-4816-9a90-032a209b4933" containerName="istio-proxy" Nov 02 23:25:13.845946 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:13.845853 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="801851c2-20cb-4816-9a90-032a209b4933" containerName="istio-proxy" Nov 02 23:25:13.846709 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:13.846000 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="801851c2-20cb-4816-9a90-032a209b4933" containerName="istio-proxy" Nov 02 23:25:13.846709 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:13.846020 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="801851c2-20cb-4816-9a90-032a209b4933" containerName="istio-proxy" Nov 02 23:25:13.922231 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:13.922191 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9"] Nov 02 23:25:13.922418 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:13.922356 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" Nov 02 23:25:14.014846 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:14.014809 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-7fdfca-openshift-default-dockercfg-tn7cf\"" Nov 02 23:25:14.020550 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:14.020493 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88hgc\" (UniqueName: \"kubernetes.io/projected/c3d41320-5841-4706-9e82-e5619986d711-kube-api-access-88hgc\") pod \"e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9\" (UID: \"c3d41320-5841-4706-9e82-e5619986d711\") " pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" Nov 02 23:25:14.021200 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:14.021157 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3d41320-5841-4706-9e82-e5619986d711-istio-podinfo\") pod \"e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9\" (UID: \"c3d41320-5841-4706-9e82-e5619986d711\") " pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" Nov 02 23:25:14.021824 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:14.021764 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3d41320-5841-4706-9e82-e5619986d711-istio-token\") pod \"e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9\" (UID: \"c3d41320-5841-4706-9e82-e5619986d711\") " pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" Nov 02 23:25:14.022176 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:14.022157 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3d41320-5841-4706-9e82-e5619986d711-istiod-ca-cert\") pod \"e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9\" (UID: \"c3d41320-5841-4706-9e82-e5619986d711\") " pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" Nov 02 23:25:14.022449 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:14.022408 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3d41320-5841-4706-9e82-e5619986d711-istio-envoy\") pod \"e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9\" (UID: \"c3d41320-5841-4706-9e82-e5619986d711\") " pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" Nov 02 23:25:14.022684 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:14.022624 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3d41320-5841-4706-9e82-e5619986d711-istio-data\") pod \"e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9\" (UID: \"c3d41320-5841-4706-9e82-e5619986d711\") " pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" Nov 02 23:25:14.022926 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:14.022907 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d41320-5841-4706-9e82-e5619986d711-credential-socket\") pod \"e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9\" (UID: \"c3d41320-5841-4706-9e82-e5619986d711\") " pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" Nov 02 23:25:14.023219 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:14.023185 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d41320-5841-4706-9e82-e5619986d711-workload-socket\") pod \"e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9\" (UID: \"c3d41320-5841-4706-9e82-e5619986d711\") " pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" Nov 02 23:25:14.023482 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:14.023429 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3d41320-5841-4706-9e82-e5619986d711-workload-certs\") pod \"e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9\" (UID: \"c3d41320-5841-4706-9e82-e5619986d711\") " pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" Nov 02 23:25:14.124514 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:14.124428 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3d41320-5841-4706-9e82-e5619986d711-istio-podinfo\") pod \"e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9\" (UID: \"c3d41320-5841-4706-9e82-e5619986d711\") " pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" Nov 02 23:25:14.124514 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:14.124469 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3d41320-5841-4706-9e82-e5619986d711-istio-token\") pod \"e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9\" (UID: \"c3d41320-5841-4706-9e82-e5619986d711\") " pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" Nov 02 23:25:14.124724 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:14.124551 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3d41320-5841-4706-9e82-e5619986d711-istiod-ca-cert\") pod \"e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9\" (UID: \"c3d41320-5841-4706-9e82-e5619986d711\") " pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" Nov 02 23:25:14.124724 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:14.124588 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3d41320-5841-4706-9e82-e5619986d711-istio-envoy\") pod \"e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9\" (UID: \"c3d41320-5841-4706-9e82-e5619986d711\") " pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" Nov 02 23:25:14.124724 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:14.124603 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3d41320-5841-4706-9e82-e5619986d711-istio-data\") pod \"e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9\" (UID: \"c3d41320-5841-4706-9e82-e5619986d711\") " pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" Nov 02 23:25:14.124724 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:14.124626 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d41320-5841-4706-9e82-e5619986d711-credential-socket\") pod \"e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9\" (UID: \"c3d41320-5841-4706-9e82-e5619986d711\") " pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" Nov 02 23:25:14.124973 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:14.124861 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d41320-5841-4706-9e82-e5619986d711-workload-socket\") pod \"e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9\" (UID: \"c3d41320-5841-4706-9e82-e5619986d711\") " pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" Nov 02 23:25:14.125038 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:14.124988 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3d41320-5841-4706-9e82-e5619986d711-workload-certs\") pod \"e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9\" (UID: \"c3d41320-5841-4706-9e82-e5619986d711\") " pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" Nov 02 23:25:14.125038 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:14.125027 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88hgc\" (UniqueName: \"kubernetes.io/projected/c3d41320-5841-4706-9e82-e5619986d711-kube-api-access-88hgc\") pod \"e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9\" (UID: \"c3d41320-5841-4706-9e82-e5619986d711\") " pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" Nov 02 23:25:14.125239 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:14.125182 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d41320-5841-4706-9e82-e5619986d711-workload-socket\") pod \"e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9\" (UID: \"c3d41320-5841-4706-9e82-e5619986d711\") " pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" Nov 02 23:25:14.125467 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:14.125439 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3d41320-5841-4706-9e82-e5619986d711-workload-certs\") pod \"e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9\" (UID: \"c3d41320-5841-4706-9e82-e5619986d711\") " pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" Nov 02 23:25:14.125548 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:14.125506 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3d41320-5841-4706-9e82-e5619986d711-istiod-ca-cert\") pod \"e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9\" (UID: \"c3d41320-5841-4706-9e82-e5619986d711\") " pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" Nov 02 23:25:14.125548 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:14.125518 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3d41320-5841-4706-9e82-e5619986d711-istio-data\") pod \"e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9\" (UID: \"c3d41320-5841-4706-9e82-e5619986d711\") " pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" Nov 02 23:25:14.125656 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:14.125598 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d41320-5841-4706-9e82-e5619986d711-credential-socket\") pod \"e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9\" (UID: \"c3d41320-5841-4706-9e82-e5619986d711\") " pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" Nov 02 23:25:14.127032 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:14.127011 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3d41320-5841-4706-9e82-e5619986d711-istio-podinfo\") pod \"e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9\" (UID: \"c3d41320-5841-4706-9e82-e5619986d711\") " pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" Nov 02 23:25:14.127278 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:14.127258 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3d41320-5841-4706-9e82-e5619986d711-istio-envoy\") pod \"e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9\" (UID: \"c3d41320-5841-4706-9e82-e5619986d711\") " pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" Nov 02 23:25:14.180151 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:14.180123 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3d41320-5841-4706-9e82-e5619986d711-istio-token\") pod \"e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9\" (UID: \"c3d41320-5841-4706-9e82-e5619986d711\") " pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" Nov 02 23:25:14.180403 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:14.180380 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88hgc\" (UniqueName: \"kubernetes.io/projected/c3d41320-5841-4706-9e82-e5619986d711-kube-api-access-88hgc\") pod \"e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9\" (UID: \"c3d41320-5841-4706-9e82-e5619986d711\") " pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" Nov 02 23:25:14.241608 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:14.241580 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" Nov 02 23:25:14.673446 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:14.673401 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9"] Nov 02 23:25:14.705903 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:25:14.705858 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3d41320_5841_4706_9e82_e5619986d711.slice/crio-7eb4ee84a1a19e332e74e9461a7d8434c646d4bb90b65ad2a8bb515577c91750 WatchSource:0}: Error finding container 7eb4ee84a1a19e332e74e9461a7d8434c646d4bb90b65ad2a8bb515577c91750: Status 404 returned error can't find the container with id 7eb4ee84a1a19e332e74e9461a7d8434c646d4bb90b65ad2a8bb515577c91750 Nov 02 23:25:14.708497 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:14.708463 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:25:14.708573 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:14.708531 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:25:14.708573 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:14.708561 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:25:15.527402 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:15.527366 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" event={"ID":"c3d41320-5841-4706-9e82-e5619986d711","Type":"ContainerStarted","Data":"0ef99bfbdc0426efb15add5364e6a19faa32f368b40c8e6e1edef5523018c00e"} Nov 02 23:25:15.528272 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:15.527435 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" event={"ID":"c3d41320-5841-4706-9e82-e5619986d711","Type":"ContainerStarted","Data":"7eb4ee84a1a19e332e74e9461a7d8434c646d4bb90b65ad2a8bb515577c91750"} Nov 02 23:25:15.659938 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:15.659862 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" podStartSLOduration=2.659847406 podStartE2EDuration="2.659847406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:25:15.656510558 +0000 UTC m=+1964.625042341" watchObservedRunningTime="2026-11-02 23:25:15.659847406 +0000 UTC m=+1964.628379189" Nov 02 23:25:16.242568 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:16.242525 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" Nov 02 23:25:16.248546 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:16.248522 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" Nov 02 23:25:16.534980 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:16.534908 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" Nov 02 23:25:16.536068 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:16.536046 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" Nov 02 23:25:21.095431 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:21.095399 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt"] Nov 02 23:25:21.095872 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:21.095636 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt" podUID="0cfd13c3-8704-4922-9a64-85882311360f" containerName="maas-api" containerID="cri-o://b9002a29f53a29d7fcfa2e61728b0cc96c34a2b61bfc1be1ba9d396f0667ab02" gracePeriod=30 Nov 02 23:25:21.412354 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:21.412328 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt" Nov 02 23:25:21.469694 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:21.469652 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cfd13c3-8704-4922-9a64-85882311360f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cfd13c3-8704-4922-9a64-85882311360f-metrics-certs\") pod \"0cfd13c3-8704-4922-9a64-85882311360f\" (UID: \"0cfd13c3-8704-4922-9a64-85882311360f\") " Nov 02 23:25:21.480052 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:21.469967 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cfd13c3-8704-4922-9a64-85882311360f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cfd13c3-8704-4922-9a64-85882311360f-maas-api-tls\") pod \"0cfd13c3-8704-4922-9a64-85882311360f\" (UID: \"0cfd13c3-8704-4922-9a64-85882311360f\") " Nov 02 23:25:21.480373 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:21.480343 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cfd13c3-8704-4922-9a64-85882311360f-kube-api-access-vc78b\" (UniqueName: \"kubernetes.io/projected/0cfd13c3-8704-4922-9a64-85882311360f-kube-api-access-vc78b\") pod \"0cfd13c3-8704-4922-9a64-85882311360f\" (UID: \"0cfd13c3-8704-4922-9a64-85882311360f\") " Nov 02 23:25:21.480596 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:21.480568 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cfd13c3-8704-4922-9a64-85882311360f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cfd13c3-8704-4922-9a64-85882311360f-openshift-service-ca\") pod \"0cfd13c3-8704-4922-9a64-85882311360f\" (UID: \"0cfd13c3-8704-4922-9a64-85882311360f\") " Nov 02 23:25:21.496748 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:21.493608 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cfd13c3-8704-4922-9a64-85882311360f-openshift-service-ca" pod "0cfd13c3-8704-4922-9a64-85882311360f" (UID: "0cfd13c3-8704-4922-9a64-85882311360f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:25:21.497076 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:21.497014 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cfd13c3-8704-4922-9a64-85882311360f-kube-api-access-vc78b" pod "0cfd13c3-8704-4922-9a64-85882311360f" (UID: "0cfd13c3-8704-4922-9a64-85882311360f"). InnerVolumeSpecName "kube-api-access-vc78b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:25:21.506928 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:21.504697 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cfd13c3-8704-4922-9a64-85882311360f-maas-api-tls" pod "0cfd13c3-8704-4922-9a64-85882311360f" (UID: "0cfd13c3-8704-4922-9a64-85882311360f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:25:21.506928 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:21.504823 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cfd13c3-8704-4922-9a64-85882311360f-metrics-certs" pod "0cfd13c3-8704-4922-9a64-85882311360f" (UID: "0cfd13c3-8704-4922-9a64-85882311360f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:25:21.567113 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:21.567030 5076 generic.go:356] "Generic (PLEG): container finished" podID="0cfd13c3-8704-4922-9a64-85882311360f" containerID="b9002a29f53a29d7fcfa2e61728b0cc96c34a2b61bfc1be1ba9d396f0667ab02" exitCode=0 Nov 02 23:25:21.567113 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:21.567075 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt" Nov 02 23:25:21.567113 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:21.567100 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt" event={"ID":"0cfd13c3-8704-4922-9a64-85882311360f","Type":"ContainerDied","Data":"b9002a29f53a29d7fcfa2e61728b0cc96c34a2b61bfc1be1ba9d396f0667ab02"} Nov 02 23:25:21.567408 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:21.567165 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt" event={"ID":"0cfd13c3-8704-4922-9a64-85882311360f","Type":"ContainerDied","Data":"93eb67247138704c7476a20cbfd9073ca0a0a2c6d9f8c16293415d7c92caca65"} Nov 02 23:25:21.567408 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:21.567198 5076 scope.go:122] "RemoveContainer" containerID="b9002a29f53a29d7fcfa2e61728b0cc96c34a2b61bfc1be1ba9d396f0667ab02" Nov 02 23:25:21.578719 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:21.578702 5076 scope.go:122] "RemoveContainer" containerID="b9002a29f53a29d7fcfa2e61728b0cc96c34a2b61bfc1be1ba9d396f0667ab02" Nov 02 23:25:21.579037 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:25:21.579018 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9002a29f53a29d7fcfa2e61728b0cc96c34a2b61bfc1be1ba9d396f0667ab02\": container with ID starting with b9002a29f53a29d7fcfa2e61728b0cc96c34a2b61bfc1be1ba9d396f0667ab02 not found: ID does not exist" containerID="b9002a29f53a29d7fcfa2e61728b0cc96c34a2b61bfc1be1ba9d396f0667ab02" Nov 02 23:25:21.579123 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:21.579052 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9002a29f53a29d7fcfa2e61728b0cc96c34a2b61bfc1be1ba9d396f0667ab02"} err="failed to get container status \"b9002a29f53a29d7fcfa2e61728b0cc96c34a2b61bfc1be1ba9d396f0667ab02\": rpc error: code = NotFound desc = could not find container \"b9002a29f53a29d7fcfa2e61728b0cc96c34a2b61bfc1be1ba9d396f0667ab02\": container with ID starting with b9002a29f53a29d7fcfa2e61728b0cc96c34a2b61bfc1be1ba9d396f0667ab02 not found: ID does not exist" Nov 02 23:25:21.582753 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:21.582733 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cfd13c3-8704-4922-9a64-85882311360f-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:21.582811 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:21.582756 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cfd13c3-8704-4922-9a64-85882311360f-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:21.582811 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:21.582767 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc78b\" (UniqueName: \"kubernetes.io/projected/0cfd13c3-8704-4922-9a64-85882311360f-kube-api-access-vc78b\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:21.582811 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:21.582777 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cfd13c3-8704-4922-9a64-85882311360f-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:21.723770 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:21.723736 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt"] Nov 02 23:25:21.730789 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:21.730758 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0064f-8d4458479-gsjjt"] Nov 02 23:25:23.148553 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:23.148521 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cfd13c3-8704-4922-9a64-85882311360f" path="/var/lib/kubelet/pods/0cfd13c3-8704-4922-9a64-85882311360f/volumes" Nov 02 23:25:28.303303 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:28.303263 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct"] Nov 02 23:25:28.303769 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:28.303505 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" podUID="c7bcd988-c4a5-46e6-bba1-c4346f21922d" containerName="istio-proxy" containerID="cri-o://a7815162a8059e9cde8df60c7aa3ef764e862f6c0567ea5b14a0942c365491a4" gracePeriod=30 Nov 02 23:25:29.752850 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:29.752811 5076 patch_prober.go:29] interesting pod/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:25:29.753261 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:29.752870 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" podUID="c7bcd988-c4a5-46e6-bba1-c4346f21922d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:25:30.266530 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:30.266496 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m"] Nov 02 23:25:30.267195 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:30.267177 5076 cpu_manager.go:414] "containerMap: removing container" podUID="0cfd13c3-8704-4922-9a64-85882311360f" containerName="maas-api" Nov 02 23:25:30.267292 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:30.267198 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cfd13c3-8704-4922-9a64-85882311360f" containerName="maas-api" Nov 02 23:25:30.267451 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:30.267430 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cfd13c3-8704-4922-9a64-85882311360f" containerName="maas-api" Nov 02 23:25:30.272288 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:30.272269 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" Nov 02 23:25:30.340674 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:30.340637 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bcab402-b870-4284-9d80-a7c8243492bd-workload-socket\") pod \"e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m\" (UID: \"8bcab402-b870-4284-9d80-a7c8243492bd\") " pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" Nov 02 23:25:30.340883 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:30.340775 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bcab402-b870-4284-9d80-a7c8243492bd-istiod-ca-cert\") pod \"e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m\" (UID: \"8bcab402-b870-4284-9d80-a7c8243492bd\") " pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" Nov 02 23:25:30.340883 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:30.340830 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bcab402-b870-4284-9d80-a7c8243492bd-istio-data\") pod \"e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m\" (UID: \"8bcab402-b870-4284-9d80-a7c8243492bd\") " pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" Nov 02 23:25:30.341009 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:30.340982 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bcab402-b870-4284-9d80-a7c8243492bd-workload-certs\") pod \"e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m\" (UID: \"8bcab402-b870-4284-9d80-a7c8243492bd\") " pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" Nov 02 23:25:30.341046 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:30.341014 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bcab402-b870-4284-9d80-a7c8243492bd-istio-envoy\") pod \"e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m\" (UID: \"8bcab402-b870-4284-9d80-a7c8243492bd\") " pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" Nov 02 23:25:30.341183 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:30.341148 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bcab402-b870-4284-9d80-a7c8243492bd-istio-token\") pod \"e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m\" (UID: \"8bcab402-b870-4284-9d80-a7c8243492bd\") " pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" Nov 02 23:25:30.341250 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:30.341214 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bcab402-b870-4284-9d80-a7c8243492bd-credential-socket\") pod \"e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m\" (UID: \"8bcab402-b870-4284-9d80-a7c8243492bd\") " pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" Nov 02 23:25:30.341309 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:30.341252 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgmzx\" (UniqueName: \"kubernetes.io/projected/8bcab402-b870-4284-9d80-a7c8243492bd-kube-api-access-bgmzx\") pod \"e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m\" (UID: \"8bcab402-b870-4284-9d80-a7c8243492bd\") " pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" Nov 02 23:25:30.341380 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:30.341366 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bcab402-b870-4284-9d80-a7c8243492bd-istio-podinfo\") pod \"e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m\" (UID: \"8bcab402-b870-4284-9d80-a7c8243492bd\") " pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" Nov 02 23:25:30.366093 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:30.366060 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m"] Nov 02 23:25:30.436921 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:30.418506 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-81c3c9-openshift-default-dockercfg-gbjcb\"" Nov 02 23:25:30.451912 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:30.447788 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bcab402-b870-4284-9d80-a7c8243492bd-istio-podinfo\") pod \"e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m\" (UID: \"8bcab402-b870-4284-9d80-a7c8243492bd\") " pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" Nov 02 23:25:30.451912 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:30.447870 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bcab402-b870-4284-9d80-a7c8243492bd-workload-socket\") pod \"e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m\" (UID: \"8bcab402-b870-4284-9d80-a7c8243492bd\") " pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" Nov 02 23:25:30.451912 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:30.447992 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bcab402-b870-4284-9d80-a7c8243492bd-istiod-ca-cert\") pod \"e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m\" (UID: \"8bcab402-b870-4284-9d80-a7c8243492bd\") " pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" Nov 02 23:25:30.451912 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:30.448035 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bcab402-b870-4284-9d80-a7c8243492bd-istio-data\") pod \"e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m\" (UID: \"8bcab402-b870-4284-9d80-a7c8243492bd\") " pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" Nov 02 23:25:30.451912 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:30.448136 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bcab402-b870-4284-9d80-a7c8243492bd-workload-certs\") pod \"e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m\" (UID: \"8bcab402-b870-4284-9d80-a7c8243492bd\") " pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" Nov 02 23:25:30.451912 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:30.448170 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bcab402-b870-4284-9d80-a7c8243492bd-istio-envoy\") pod \"e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m\" (UID: \"8bcab402-b870-4284-9d80-a7c8243492bd\") " pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" Nov 02 23:25:30.451912 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:30.448266 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bcab402-b870-4284-9d80-a7c8243492bd-istio-token\") pod \"e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m\" (UID: \"8bcab402-b870-4284-9d80-a7c8243492bd\") " pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" Nov 02 23:25:30.451912 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:30.448305 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bcab402-b870-4284-9d80-a7c8243492bd-credential-socket\") pod \"e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m\" (UID: \"8bcab402-b870-4284-9d80-a7c8243492bd\") " pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" Nov 02 23:25:30.451912 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:30.448330 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgmzx\" (UniqueName: \"kubernetes.io/projected/8bcab402-b870-4284-9d80-a7c8243492bd-kube-api-access-bgmzx\") pod \"e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m\" (UID: \"8bcab402-b870-4284-9d80-a7c8243492bd\") " pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" Nov 02 23:25:30.451912 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:30.448489 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bcab402-b870-4284-9d80-a7c8243492bd-workload-socket\") pod \"e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m\" (UID: \"8bcab402-b870-4284-9d80-a7c8243492bd\") " pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" Nov 02 23:25:30.454135 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:30.454097 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bcab402-b870-4284-9d80-a7c8243492bd-istiod-ca-cert\") pod \"e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m\" (UID: \"8bcab402-b870-4284-9d80-a7c8243492bd\") " pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" Nov 02 23:25:30.473911 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:30.470584 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bcab402-b870-4284-9d80-a7c8243492bd-istio-data\") pod \"e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m\" (UID: \"8bcab402-b870-4284-9d80-a7c8243492bd\") " pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" Nov 02 23:25:30.474746 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:30.474682 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bcab402-b870-4284-9d80-a7c8243492bd-istio-podinfo\") pod \"e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m\" (UID: \"8bcab402-b870-4284-9d80-a7c8243492bd\") " pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" Nov 02 23:25:30.496912 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:30.480726 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bcab402-b870-4284-9d80-a7c8243492bd-credential-socket\") pod \"e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m\" (UID: \"8bcab402-b870-4284-9d80-a7c8243492bd\") " pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" Nov 02 23:25:30.496912 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:30.486407 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bcab402-b870-4284-9d80-a7c8243492bd-workload-certs\") pod \"e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m\" (UID: \"8bcab402-b870-4284-9d80-a7c8243492bd\") " pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" Nov 02 23:25:30.512464 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:30.512396 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bcab402-b870-4284-9d80-a7c8243492bd-istio-envoy\") pod \"e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m\" (UID: \"8bcab402-b870-4284-9d80-a7c8243492bd\") " pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" Nov 02 23:25:30.514713 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:30.514646 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgmzx\" (UniqueName: \"kubernetes.io/projected/8bcab402-b870-4284-9d80-a7c8243492bd-kube-api-access-bgmzx\") pod \"e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m\" (UID: \"8bcab402-b870-4284-9d80-a7c8243492bd\") " pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" Nov 02 23:25:30.594766 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:30.594691 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bcab402-b870-4284-9d80-a7c8243492bd-istio-token\") pod \"e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m\" (UID: \"8bcab402-b870-4284-9d80-a7c8243492bd\") " pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" Nov 02 23:25:30.896122 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:30.896034 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" Nov 02 23:25:31.316159 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:31.316112 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m"] Nov 02 23:25:31.317339 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:25:31.317313 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bcab402_b870_4284_9d80_a7c8243492bd.slice/crio-46456ba57e22fda260e794ae84fac28e0c995ef5c3e274d2b7a6e8efc12453c1 WatchSource:0}: Error finding container 46456ba57e22fda260e794ae84fac28e0c995ef5c3e274d2b7a6e8efc12453c1: Status 404 returned error can't find the container with id 46456ba57e22fda260e794ae84fac28e0c995ef5c3e274d2b7a6e8efc12453c1 Nov 02 23:25:31.319951 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:31.319920 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:25:31.320019 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:31.319985 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:25:31.320019 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:31.320015 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:25:31.637884 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:31.637800 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" event={"ID":"8bcab402-b870-4284-9d80-a7c8243492bd","Type":"ContainerStarted","Data":"8c6b87f5d8752b4cfb609abe74bb9468ddeaedd3545e3d93c816bf11b3814cc4"} Nov 02 23:25:31.637884 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:31.637858 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" event={"ID":"8bcab402-b870-4284-9d80-a7c8243492bd","Type":"ContainerStarted","Data":"46456ba57e22fda260e794ae84fac28e0c995ef5c3e274d2b7a6e8efc12453c1"} Nov 02 23:25:31.752079 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:31.752020 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" podStartSLOduration=1.752005477 podStartE2EDuration="1.752005477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:25:31.749014672 +0000 UTC m=+1980.717546456" watchObservedRunningTime="2026-11-02 23:25:31.752005477 +0000 UTC m=+1980.720537261" Nov 02 23:25:31.892572 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:31.889137 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" Nov 02 23:25:32.993769 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:32.993739 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" Nov 02 23:25:33.590783 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.590753 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" Nov 02 23:25:33.653254 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.653217 5076 generic.go:356] "Generic (PLEG): container finished" podID="c7bcd988-c4a5-46e6-bba1-c4346f21922d" containerID="a7815162a8059e9cde8df60c7aa3ef764e862f6c0567ea5b14a0942c365491a4" exitCode=0 Nov 02 23:25:33.653420 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.653274 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" Nov 02 23:25:33.653420 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.653296 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" event={"ID":"c7bcd988-c4a5-46e6-bba1-c4346f21922d","Type":"ContainerDied","Data":"a7815162a8059e9cde8df60c7aa3ef764e862f6c0567ea5b14a0942c365491a4"} Nov 02 23:25:33.653420 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.653350 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct" event={"ID":"c7bcd988-c4a5-46e6-bba1-c4346f21922d","Type":"ContainerDied","Data":"af957e6f639869a01e32d0b8a3fbf78930b7837f662431c122a76adf09aa13f0"} Nov 02 23:25:33.653420 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.653371 5076 scope.go:122] "RemoveContainer" containerID="a7815162a8059e9cde8df60c7aa3ef764e862f6c0567ea5b14a0942c365491a4" Nov 02 23:25:33.653630 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.653601 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" Nov 02 23:25:33.654714 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.654697 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" Nov 02 23:25:33.663528 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.663504 5076 scope.go:122] "RemoveContainer" containerID="3610146349edc6f65d4cd50dd26f467e09bb2d807ae0606d3e6dfd74516329f7" Nov 02 23:25:33.672630 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.672603 5076 scope.go:122] "RemoveContainer" containerID="a7815162a8059e9cde8df60c7aa3ef764e862f6c0567ea5b14a0942c365491a4" Nov 02 23:25:33.672875 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:25:33.672856 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7815162a8059e9cde8df60c7aa3ef764e862f6c0567ea5b14a0942c365491a4\": container with ID starting with a7815162a8059e9cde8df60c7aa3ef764e862f6c0567ea5b14a0942c365491a4 not found: ID does not exist" containerID="a7815162a8059e9cde8df60c7aa3ef764e862f6c0567ea5b14a0942c365491a4" Nov 02 23:25:33.672941 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.672882 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7815162a8059e9cde8df60c7aa3ef764e862f6c0567ea5b14a0942c365491a4"} err="failed to get container status \"a7815162a8059e9cde8df60c7aa3ef764e862f6c0567ea5b14a0942c365491a4\": rpc error: code = NotFound desc = could not find container \"a7815162a8059e9cde8df60c7aa3ef764e862f6c0567ea5b14a0942c365491a4\": container with ID starting with a7815162a8059e9cde8df60c7aa3ef764e862f6c0567ea5b14a0942c365491a4 not found: ID does not exist" Nov 02 23:25:33.672941 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.672920 5076 scope.go:122] "RemoveContainer" containerID="3610146349edc6f65d4cd50dd26f467e09bb2d807ae0606d3e6dfd74516329f7" Nov 02 23:25:33.673139 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:25:33.673120 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3610146349edc6f65d4cd50dd26f467e09bb2d807ae0606d3e6dfd74516329f7\": container with ID starting with 3610146349edc6f65d4cd50dd26f467e09bb2d807ae0606d3e6dfd74516329f7 not found: ID does not exist" containerID="3610146349edc6f65d4cd50dd26f467e09bb2d807ae0606d3e6dfd74516329f7" Nov 02 23:25:33.673182 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.673143 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3610146349edc6f65d4cd50dd26f467e09bb2d807ae0606d3e6dfd74516329f7"} err="failed to get container status \"3610146349edc6f65d4cd50dd26f467e09bb2d807ae0606d3e6dfd74516329f7\": rpc error: code = NotFound desc = could not find container \"3610146349edc6f65d4cd50dd26f467e09bb2d807ae0606d3e6dfd74516329f7\": container with ID starting with 3610146349edc6f65d4cd50dd26f467e09bb2d807ae0606d3e6dfd74516329f7 not found: ID does not exist" Nov 02 23:25:33.710729 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.710691 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7bcd988-c4a5-46e6-bba1-c4346f21922d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7bcd988-c4a5-46e6-bba1-c4346f21922d-workload-socket\") pod \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\" (UID: \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\") " Nov 02 23:25:33.710900 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.710781 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7bcd988-c4a5-46e6-bba1-c4346f21922d-kube-api-access-4sfr2\" (UniqueName: \"kubernetes.io/projected/c7bcd988-c4a5-46e6-bba1-c4346f21922d-kube-api-access-4sfr2\") pod \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\" (UID: \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\") " Nov 02 23:25:33.710900 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.710832 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7bcd988-c4a5-46e6-bba1-c4346f21922d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7bcd988-c4a5-46e6-bba1-c4346f21922d-credential-socket\") pod \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\" (UID: \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\") " Nov 02 23:25:33.711037 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.710961 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7bcd988-c4a5-46e6-bba1-c4346f21922d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7bcd988-c4a5-46e6-bba1-c4346f21922d-istio-podinfo\") pod \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\" (UID: \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\") " Nov 02 23:25:33.711037 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.711002 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7bcd988-c4a5-46e6-bba1-c4346f21922d-istio-token\" (UniqueName: \"kubernetes.io/projected/c7bcd988-c4a5-46e6-bba1-c4346f21922d-istio-token\") pod \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\" (UID: \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\") " Nov 02 23:25:33.711152 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.711029 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7bcd988-c4a5-46e6-bba1-c4346f21922d-workload-socket" pod "c7bcd988-c4a5-46e6-bba1-c4346f21922d" (UID: "c7bcd988-c4a5-46e6-bba1-c4346f21922d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:33.711152 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.711109 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7bcd988-c4a5-46e6-bba1-c4346f21922d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7bcd988-c4a5-46e6-bba1-c4346f21922d-workload-certs\") pod \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\" (UID: \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\") " Nov 02 23:25:33.711257 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.711158 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7bcd988-c4a5-46e6-bba1-c4346f21922d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7bcd988-c4a5-46e6-bba1-c4346f21922d-istiod-ca-cert\") pod \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\" (UID: \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\") " Nov 02 23:25:33.711257 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.711204 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7bcd988-c4a5-46e6-bba1-c4346f21922d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7bcd988-c4a5-46e6-bba1-c4346f21922d-istio-envoy\") pod \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\" (UID: \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\") " Nov 02 23:25:33.711867 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.711494 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7bcd988-c4a5-46e6-bba1-c4346f21922d-workload-certs" pod "c7bcd988-c4a5-46e6-bba1-c4346f21922d" (UID: "c7bcd988-c4a5-46e6-bba1-c4346f21922d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:33.711867 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.711543 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7bcd988-c4a5-46e6-bba1-c4346f21922d-credential-socket" pod "c7bcd988-c4a5-46e6-bba1-c4346f21922d" (UID: "c7bcd988-c4a5-46e6-bba1-c4346f21922d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:33.711867 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.711659 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7bcd988-c4a5-46e6-bba1-c4346f21922d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7bcd988-c4a5-46e6-bba1-c4346f21922d-istio-data\") pod \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\" (UID: \"c7bcd988-c4a5-46e6-bba1-c4346f21922d\") " Nov 02 23:25:33.711867 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.711823 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7bcd988-c4a5-46e6-bba1-c4346f21922d-istiod-ca-cert" pod "c7bcd988-c4a5-46e6-bba1-c4346f21922d" (UID: "c7bcd988-c4a5-46e6-bba1-c4346f21922d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:25:33.713073 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.712447 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7bcd988-c4a5-46e6-bba1-c4346f21922d-istio-data" pod "c7bcd988-c4a5-46e6-bba1-c4346f21922d" (UID: "c7bcd988-c4a5-46e6-bba1-c4346f21922d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:33.713073 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.712475 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7bcd988-c4a5-46e6-bba1-c4346f21922d-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:33.713073 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.712499 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7bcd988-c4a5-46e6-bba1-c4346f21922d-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:33.713073 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.712515 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7bcd988-c4a5-46e6-bba1-c4346f21922d-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:33.713073 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.712529 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7bcd988-c4a5-46e6-bba1-c4346f21922d-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:33.713746 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.713724 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7bcd988-c4a5-46e6-bba1-c4346f21922d-kube-api-access-4sfr2" pod "c7bcd988-c4a5-46e6-bba1-c4346f21922d" (UID: "c7bcd988-c4a5-46e6-bba1-c4346f21922d"). InnerVolumeSpecName "kube-api-access-4sfr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:25:33.715033 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.715006 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7bcd988-c4a5-46e6-bba1-c4346f21922d-istio-token" pod "c7bcd988-c4a5-46e6-bba1-c4346f21922d" (UID: "c7bcd988-c4a5-46e6-bba1-c4346f21922d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:25:33.715263 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.715234 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7bcd988-c4a5-46e6-bba1-c4346f21922d-istio-podinfo" pod "c7bcd988-c4a5-46e6-bba1-c4346f21922d" (UID: "c7bcd988-c4a5-46e6-bba1-c4346f21922d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:25:33.715491 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.715468 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7bcd988-c4a5-46e6-bba1-c4346f21922d-istio-envoy" pod "c7bcd988-c4a5-46e6-bba1-c4346f21922d" (UID: "c7bcd988-c4a5-46e6-bba1-c4346f21922d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:33.813598 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.813568 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7bcd988-c4a5-46e6-bba1-c4346f21922d-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:33.813749 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.813594 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7bcd988-c4a5-46e6-bba1-c4346f21922d-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:33.813749 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.813619 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7bcd988-c4a5-46e6-bba1-c4346f21922d-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:33.813749 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.813627 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7bcd988-c4a5-46e6-bba1-c4346f21922d-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:33.813749 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:33.813638 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sfr2\" (UniqueName: \"kubernetes.io/projected/c7bcd988-c4a5-46e6-bba1-c4346f21922d-kube-api-access-4sfr2\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:34.072015 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:34.070652 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct"] Nov 02 23:25:34.074131 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:34.074107 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-c0064f-openshift-default-7d857694d7-cfhct"] Nov 02 23:25:35.156457 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:35.156409 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7bcd988-c4a5-46e6-bba1-c4346f21922d" path="/var/lib/kubelet/pods/c7bcd988-c4a5-46e6-bba1-c4346f21922d/volumes" Nov 02 23:25:38.157288 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:38.157251 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8"] Nov 02 23:25:38.157738 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:38.157463 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8" podUID="11c1048d-56a0-4abf-97a0-f0b99ed82d7c" containerName="maas-api" containerID="cri-o://09e2017428c0e542bd25bc2a2f06563ef13ea23600f38525af80fe0f2286b23b" gracePeriod=30 Nov 02 23:25:38.428232 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:38.428205 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8" Nov 02 23:25:38.580100 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:38.580062 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11c1048d-56a0-4abf-97a0-f0b99ed82d7c-kube-api-access-dtst8\" (UniqueName: \"kubernetes.io/projected/11c1048d-56a0-4abf-97a0-f0b99ed82d7c-kube-api-access-dtst8\") pod \"11c1048d-56a0-4abf-97a0-f0b99ed82d7c\" (UID: \"11c1048d-56a0-4abf-97a0-f0b99ed82d7c\") " Nov 02 23:25:38.580303 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:38.580116 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11c1048d-56a0-4abf-97a0-f0b99ed82d7c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11c1048d-56a0-4abf-97a0-f0b99ed82d7c-maas-api-tls\") pod \"11c1048d-56a0-4abf-97a0-f0b99ed82d7c\" (UID: \"11c1048d-56a0-4abf-97a0-f0b99ed82d7c\") " Nov 02 23:25:38.580303 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:38.580185 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11c1048d-56a0-4abf-97a0-f0b99ed82d7c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/11c1048d-56a0-4abf-97a0-f0b99ed82d7c-metrics-certs\") pod \"11c1048d-56a0-4abf-97a0-f0b99ed82d7c\" (UID: \"11c1048d-56a0-4abf-97a0-f0b99ed82d7c\") " Nov 02 23:25:38.580303 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:38.580221 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11c1048d-56a0-4abf-97a0-f0b99ed82d7c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11c1048d-56a0-4abf-97a0-f0b99ed82d7c-openshift-service-ca\") pod \"11c1048d-56a0-4abf-97a0-f0b99ed82d7c\" (UID: \"11c1048d-56a0-4abf-97a0-f0b99ed82d7c\") " Nov 02 23:25:38.580725 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:38.580696 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11c1048d-56a0-4abf-97a0-f0b99ed82d7c-openshift-service-ca" pod "11c1048d-56a0-4abf-97a0-f0b99ed82d7c" (UID: "11c1048d-56a0-4abf-97a0-f0b99ed82d7c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:25:38.582268 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:38.582239 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11c1048d-56a0-4abf-97a0-f0b99ed82d7c-metrics-certs" pod "11c1048d-56a0-4abf-97a0-f0b99ed82d7c" (UID: "11c1048d-56a0-4abf-97a0-f0b99ed82d7c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:25:38.582496 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:38.582470 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c1048d-56a0-4abf-97a0-f0b99ed82d7c-kube-api-access-dtst8" pod "11c1048d-56a0-4abf-97a0-f0b99ed82d7c" (UID: "11c1048d-56a0-4abf-97a0-f0b99ed82d7c"). InnerVolumeSpecName "kube-api-access-dtst8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:25:38.582616 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:38.582596 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11c1048d-56a0-4abf-97a0-f0b99ed82d7c-maas-api-tls" pod "11c1048d-56a0-4abf-97a0-f0b99ed82d7c" (UID: "11c1048d-56a0-4abf-97a0-f0b99ed82d7c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:25:38.681822 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:38.681738 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtst8\" (UniqueName: \"kubernetes.io/projected/11c1048d-56a0-4abf-97a0-f0b99ed82d7c-kube-api-access-dtst8\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:38.681822 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:38.681773 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11c1048d-56a0-4abf-97a0-f0b99ed82d7c-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:38.681822 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:38.681783 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11c1048d-56a0-4abf-97a0-f0b99ed82d7c-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:38.681822 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:38.681792 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11c1048d-56a0-4abf-97a0-f0b99ed82d7c-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:38.688915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:38.688860 5076 generic.go:356] "Generic (PLEG): container finished" podID="11c1048d-56a0-4abf-97a0-f0b99ed82d7c" containerID="09e2017428c0e542bd25bc2a2f06563ef13ea23600f38525af80fe0f2286b23b" exitCode=0 Nov 02 23:25:38.689051 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:38.688927 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8" Nov 02 23:25:38.689051 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:38.688936 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8" event={"ID":"11c1048d-56a0-4abf-97a0-f0b99ed82d7c","Type":"ContainerDied","Data":"09e2017428c0e542bd25bc2a2f06563ef13ea23600f38525af80fe0f2286b23b"} Nov 02 23:25:38.689051 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:38.688997 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8" event={"ID":"11c1048d-56a0-4abf-97a0-f0b99ed82d7c","Type":"ContainerDied","Data":"94b37f5ae412f0f6caf74f830ce8c63b391331429894eacb2c21cece3ff362c2"} Nov 02 23:25:38.689051 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:38.689022 5076 scope.go:122] "RemoveContainer" containerID="09e2017428c0e542bd25bc2a2f06563ef13ea23600f38525af80fe0f2286b23b" Nov 02 23:25:38.700672 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:38.700653 5076 scope.go:122] "RemoveContainer" containerID="09e2017428c0e542bd25bc2a2f06563ef13ea23600f38525af80fe0f2286b23b" Nov 02 23:25:38.700955 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:25:38.700934 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09e2017428c0e542bd25bc2a2f06563ef13ea23600f38525af80fe0f2286b23b\": container with ID starting with 09e2017428c0e542bd25bc2a2f06563ef13ea23600f38525af80fe0f2286b23b not found: ID does not exist" containerID="09e2017428c0e542bd25bc2a2f06563ef13ea23600f38525af80fe0f2286b23b" Nov 02 23:25:38.701043 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:38.700962 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09e2017428c0e542bd25bc2a2f06563ef13ea23600f38525af80fe0f2286b23b"} err="failed to get container status \"09e2017428c0e542bd25bc2a2f06563ef13ea23600f38525af80fe0f2286b23b\": rpc error: code = NotFound desc = could not find container \"09e2017428c0e542bd25bc2a2f06563ef13ea23600f38525af80fe0f2286b23b\": container with ID starting with 09e2017428c0e542bd25bc2a2f06563ef13ea23600f38525af80fe0f2286b23b not found: ID does not exist" Nov 02 23:25:38.718538 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:38.718516 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a69f65-6f9d977d6d-jcxjn"] Nov 02 23:25:38.718723 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:38.718705 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a69f65-6f9d977d6d-jcxjn" podUID="428ec111-bd83-465d-b67e-e3ffcfdc946c" containerName="payload-processing" containerID="cri-o://b49d83af05e72ba178d9a1e9d4657fe08241b179e9c29df695b83be503998ae4" gracePeriod=30 Nov 02 23:25:38.770306 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:38.770269 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-77cd547f9f-tmkt6"] Nov 02 23:25:38.770477 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:38.770462 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-77cd547f9f-tmkt6" podUID="3df93a02-4d85-4282-891a-fbb18a0e7e33" containerName="payload-pre-processing" containerID="cri-o://b2bfcf4f756bc5cc5e8c58e91714c641d283e148b6c38914c9fae5eac16969ad" gracePeriod=30 Nov 02 23:25:38.959998 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:38.959958 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8"] Nov 02 23:25:38.961877 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:38.961848 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a69f65-5545949946-7bfn8"] Nov 02 23:25:39.035691 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:39.035662 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a69f65-6f9d977d6d-jcxjn" Nov 02 23:25:39.084455 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:39.084421 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-77cd547f9f-tmkt6" Nov 02 23:25:39.148984 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:39.148950 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11c1048d-56a0-4abf-97a0-f0b99ed82d7c" path="/var/lib/kubelet/pods/11c1048d-56a0-4abf-97a0-f0b99ed82d7c/volumes" Nov 02 23:25:39.187230 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:39.187198 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/428ec111-bd83-465d-b67e-e3ffcfdc946c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428ec111-bd83-465d-b67e-e3ffcfdc946c-plugins-config-volume\") pod \"428ec111-bd83-465d-b67e-e3ffcfdc946c\" (UID: \"428ec111-bd83-465d-b67e-e3ffcfdc946c\") " Nov 02 23:25:39.188000 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:39.187970 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3df93a02-4d85-4282-891a-fbb18a0e7e33-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3df93a02-4d85-4282-891a-fbb18a0e7e33-plugins-config-volume\") pod \"3df93a02-4d85-4282-891a-fbb18a0e7e33\" (UID: \"3df93a02-4d85-4282-891a-fbb18a0e7e33\") " Nov 02 23:25:39.188103 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:39.187764 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/428ec111-bd83-465d-b67e-e3ffcfdc946c-plugins-config-volume" pod "428ec111-bd83-465d-b67e-e3ffcfdc946c" (UID: "428ec111-bd83-465d-b67e-e3ffcfdc946c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:25:39.188103 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:39.188065 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3df93a02-4d85-4282-891a-fbb18a0e7e33-kube-api-access-l87gh\" (UniqueName: \"kubernetes.io/projected/3df93a02-4d85-4282-891a-fbb18a0e7e33-kube-api-access-l87gh\") pod \"3df93a02-4d85-4282-891a-fbb18a0e7e33\" (UID: \"3df93a02-4d85-4282-891a-fbb18a0e7e33\") " Nov 02 23:25:39.188214 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:39.188203 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/428ec111-bd83-465d-b67e-e3ffcfdc946c-kube-api-access-qnxdn\" (UniqueName: \"kubernetes.io/projected/428ec111-bd83-465d-b67e-e3ffcfdc946c-kube-api-access-qnxdn\") pod \"428ec111-bd83-465d-b67e-e3ffcfdc946c\" (UID: \"428ec111-bd83-465d-b67e-e3ffcfdc946c\") " Nov 02 23:25:39.188846 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:39.188526 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3df93a02-4d85-4282-891a-fbb18a0e7e33-plugins-config-volume" pod "3df93a02-4d85-4282-891a-fbb18a0e7e33" (UID: "3df93a02-4d85-4282-891a-fbb18a0e7e33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:25:39.189092 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:39.189045 5076 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428ec111-bd83-465d-b67e-e3ffcfdc946c-plugins-config-volume\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:39.189092 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:39.189067 5076 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3df93a02-4d85-4282-891a-fbb18a0e7e33-plugins-config-volume\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:39.191612 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:39.191586 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/428ec111-bd83-465d-b67e-e3ffcfdc946c-kube-api-access-qnxdn" pod "428ec111-bd83-465d-b67e-e3ffcfdc946c" (UID: "428ec111-bd83-465d-b67e-e3ffcfdc946c"). InnerVolumeSpecName "kube-api-access-qnxdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:25:39.196478 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:39.196436 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3df93a02-4d85-4282-891a-fbb18a0e7e33-kube-api-access-l87gh" pod "3df93a02-4d85-4282-891a-fbb18a0e7e33" (UID: "3df93a02-4d85-4282-891a-fbb18a0e7e33"). InnerVolumeSpecName "kube-api-access-l87gh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:25:39.290040 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:39.290009 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l87gh\" (UniqueName: \"kubernetes.io/projected/3df93a02-4d85-4282-891a-fbb18a0e7e33-kube-api-access-l87gh\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:39.290040 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:39.290035 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnxdn\" (UniqueName: \"kubernetes.io/projected/428ec111-bd83-465d-b67e-e3ffcfdc946c-kube-api-access-qnxdn\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:39.697178 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:39.697145 5076 generic.go:356] "Generic (PLEG): container finished" podID="428ec111-bd83-465d-b67e-e3ffcfdc946c" containerID="b49d83af05e72ba178d9a1e9d4657fe08241b179e9c29df695b83be503998ae4" exitCode=0 Nov 02 23:25:39.697364 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:39.697190 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a69f65-6f9d977d6d-jcxjn" Nov 02 23:25:39.697364 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:39.697224 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a69f65-6f9d977d6d-jcxjn" event={"ID":"428ec111-bd83-465d-b67e-e3ffcfdc946c","Type":"ContainerDied","Data":"b49d83af05e72ba178d9a1e9d4657fe08241b179e9c29df695b83be503998ae4"} Nov 02 23:25:39.697364 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:39.697278 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a69f65-6f9d977d6d-jcxjn" event={"ID":"428ec111-bd83-465d-b67e-e3ffcfdc946c","Type":"ContainerDied","Data":"ea9da1527d40aa74ed07e7c1cbcfaaa1ba4aca415fa246b8d511aff29e94f21a"} Nov 02 23:25:39.697364 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:39.697294 5076 scope.go:122] "RemoveContainer" containerID="b49d83af05e72ba178d9a1e9d4657fe08241b179e9c29df695b83be503998ae4" Nov 02 23:25:39.700112 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:39.700084 5076 generic.go:356] "Generic (PLEG): container finished" podID="3df93a02-4d85-4282-891a-fbb18a0e7e33" containerID="b2bfcf4f756bc5cc5e8c58e91714c641d283e148b6c38914c9fae5eac16969ad" exitCode=0 Nov 02 23:25:39.700228 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:39.700123 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-77cd547f9f-tmkt6" event={"ID":"3df93a02-4d85-4282-891a-fbb18a0e7e33","Type":"ContainerDied","Data":"b2bfcf4f756bc5cc5e8c58e91714c641d283e148b6c38914c9fae5eac16969ad"} Nov 02 23:25:39.700228 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:39.700163 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-77cd547f9f-tmkt6" event={"ID":"3df93a02-4d85-4282-891a-fbb18a0e7e33","Type":"ContainerDied","Data":"bba83565f94e5d43cc5e6cf85257a32e233c6750535ca8fa8fd556a97a176415"} Nov 02 23:25:39.700301 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:39.700230 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-77cd547f9f-tmkt6" Nov 02 23:25:39.711690 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:39.711664 5076 scope.go:122] "RemoveContainer" containerID="b49d83af05e72ba178d9a1e9d4657fe08241b179e9c29df695b83be503998ae4" Nov 02 23:25:39.720724 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:25:39.712095 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b49d83af05e72ba178d9a1e9d4657fe08241b179e9c29df695b83be503998ae4\": container with ID starting with b49d83af05e72ba178d9a1e9d4657fe08241b179e9c29df695b83be503998ae4 not found: ID does not exist" containerID="b49d83af05e72ba178d9a1e9d4657fe08241b179e9c29df695b83be503998ae4" Nov 02 23:25:39.720724 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:39.712139 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b49d83af05e72ba178d9a1e9d4657fe08241b179e9c29df695b83be503998ae4"} err="failed to get container status \"b49d83af05e72ba178d9a1e9d4657fe08241b179e9c29df695b83be503998ae4\": rpc error: code = NotFound desc = could not find container \"b49d83af05e72ba178d9a1e9d4657fe08241b179e9c29df695b83be503998ae4\": container with ID starting with b49d83af05e72ba178d9a1e9d4657fe08241b179e9c29df695b83be503998ae4 not found: ID does not exist" Nov 02 23:25:39.720724 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:39.712163 5076 scope.go:122] "RemoveContainer" containerID="b2bfcf4f756bc5cc5e8c58e91714c641d283e148b6c38914c9fae5eac16969ad" Nov 02 23:25:39.728497 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:39.728478 5076 scope.go:122] "RemoveContainer" containerID="b2bfcf4f756bc5cc5e8c58e91714c641d283e148b6c38914c9fae5eac16969ad" Nov 02 23:25:39.728810 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:25:39.728780 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2bfcf4f756bc5cc5e8c58e91714c641d283e148b6c38914c9fae5eac16969ad\": container with ID starting with b2bfcf4f756bc5cc5e8c58e91714c641d283e148b6c38914c9fae5eac16969ad not found: ID does not exist" containerID="b2bfcf4f756bc5cc5e8c58e91714c641d283e148b6c38914c9fae5eac16969ad" Nov 02 23:25:39.728945 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:39.728812 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2bfcf4f756bc5cc5e8c58e91714c641d283e148b6c38914c9fae5eac16969ad"} err="failed to get container status \"b2bfcf4f756bc5cc5e8c58e91714c641d283e148b6c38914c9fae5eac16969ad\": rpc error: code = NotFound desc = could not find container \"b2bfcf4f756bc5cc5e8c58e91714c641d283e148b6c38914c9fae5eac16969ad\": container with ID starting with b2bfcf4f756bc5cc5e8c58e91714c641d283e148b6c38914c9fae5eac16969ad not found: ID does not exist" Nov 02 23:25:39.853982 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:39.853948 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a69f65-6f9d977d6d-jcxjn"] Nov 02 23:25:39.857853 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:39.857824 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a69f65-6f9d977d6d-jcxjn"] Nov 02 23:25:39.884702 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:25:39.884665 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:25:39.960480 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:39.960400 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-77cd547f9f-tmkt6"] Nov 02 23:25:39.964420 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:39.964394 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-77cd547f9f-tmkt6"] Nov 02 23:25:41.148188 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:41.148155 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3df93a02-4d85-4282-891a-fbb18a0e7e33" path="/var/lib/kubelet/pods/3df93a02-4d85-4282-891a-fbb18a0e7e33/volumes" Nov 02 23:25:41.148664 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:41.148565 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="428ec111-bd83-465d-b67e-e3ffcfdc946c" path="/var/lib/kubelet/pods/428ec111-bd83-465d-b67e-e3ffcfdc946c/volumes" Nov 02 23:25:44.858715 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:44.858678 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729"] Nov 02 23:25:44.859166 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:44.858938 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" podUID="dd184b6f-3901-413f-ad6a-c7271183796f" containerName="istio-proxy" containerID="cri-o://892a4ab283b16a8a62d8b6ad337a6d06a446b89f94647fd4c89beedca60ff4fe" gracePeriod=30 Nov 02 23:25:50.120317 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:50.120258 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:25:50.169773 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:50.169738 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd184b6f-3901-413f-ad6a-c7271183796f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd184b6f-3901-413f-ad6a-c7271183796f-workload-socket\") pod \"dd184b6f-3901-413f-ad6a-c7271183796f\" (UID: \"dd184b6f-3901-413f-ad6a-c7271183796f\") " Nov 02 23:25:50.169988 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:50.169801 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd184b6f-3901-413f-ad6a-c7271183796f-istio-token\" (UniqueName: \"kubernetes.io/projected/dd184b6f-3901-413f-ad6a-c7271183796f-istio-token\") pod \"dd184b6f-3901-413f-ad6a-c7271183796f\" (UID: \"dd184b6f-3901-413f-ad6a-c7271183796f\") " Nov 02 23:25:50.169988 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:50.169839 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd184b6f-3901-413f-ad6a-c7271183796f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd184b6f-3901-413f-ad6a-c7271183796f-credential-socket\") pod \"dd184b6f-3901-413f-ad6a-c7271183796f\" (UID: \"dd184b6f-3901-413f-ad6a-c7271183796f\") " Nov 02 23:25:50.169988 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:50.169931 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd184b6f-3901-413f-ad6a-c7271183796f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd184b6f-3901-413f-ad6a-c7271183796f-istio-podinfo\") pod \"dd184b6f-3901-413f-ad6a-c7271183796f\" (UID: \"dd184b6f-3901-413f-ad6a-c7271183796f\") " Nov 02 23:25:50.169988 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:50.169978 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd184b6f-3901-413f-ad6a-c7271183796f-kube-api-access-lp72b\" (UniqueName: \"kubernetes.io/projected/dd184b6f-3901-413f-ad6a-c7271183796f-kube-api-access-lp72b\") pod \"dd184b6f-3901-413f-ad6a-c7271183796f\" (UID: \"dd184b6f-3901-413f-ad6a-c7271183796f\") " Nov 02 23:25:50.170227 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:50.170059 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd184b6f-3901-413f-ad6a-c7271183796f-workload-socket" pod "dd184b6f-3901-413f-ad6a-c7271183796f" (UID: "dd184b6f-3901-413f-ad6a-c7271183796f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:50.170227 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:50.170062 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd184b6f-3901-413f-ad6a-c7271183796f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd184b6f-3901-413f-ad6a-c7271183796f-istio-data\") pod \"dd184b6f-3901-413f-ad6a-c7271183796f\" (UID: \"dd184b6f-3901-413f-ad6a-c7271183796f\") " Nov 02 23:25:50.170227 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:50.170193 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd184b6f-3901-413f-ad6a-c7271183796f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd184b6f-3901-413f-ad6a-c7271183796f-istio-envoy\") pod \"dd184b6f-3901-413f-ad6a-c7271183796f\" (UID: \"dd184b6f-3901-413f-ad6a-c7271183796f\") " Nov 02 23:25:50.170448 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:50.170222 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd184b6f-3901-413f-ad6a-c7271183796f-credential-socket" pod "dd184b6f-3901-413f-ad6a-c7271183796f" (UID: "dd184b6f-3901-413f-ad6a-c7271183796f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:50.170448 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:50.170233 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd184b6f-3901-413f-ad6a-c7271183796f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd184b6f-3901-413f-ad6a-c7271183796f-istiod-ca-cert\") pod \"dd184b6f-3901-413f-ad6a-c7271183796f\" (UID: \"dd184b6f-3901-413f-ad6a-c7271183796f\") " Nov 02 23:25:50.170448 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:50.170241 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd184b6f-3901-413f-ad6a-c7271183796f-istio-data" pod "dd184b6f-3901-413f-ad6a-c7271183796f" (UID: "dd184b6f-3901-413f-ad6a-c7271183796f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:50.170448 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:50.170324 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd184b6f-3901-413f-ad6a-c7271183796f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd184b6f-3901-413f-ad6a-c7271183796f-workload-certs\") pod \"dd184b6f-3901-413f-ad6a-c7271183796f\" (UID: \"dd184b6f-3901-413f-ad6a-c7271183796f\") " Nov 02 23:25:50.170659 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:50.170643 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd184b6f-3901-413f-ad6a-c7271183796f-istiod-ca-cert" pod "dd184b6f-3901-413f-ad6a-c7271183796f" (UID: "dd184b6f-3901-413f-ad6a-c7271183796f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:25:50.170659 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:50.170646 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd184b6f-3901-413f-ad6a-c7271183796f-workload-certs" pod "dd184b6f-3901-413f-ad6a-c7271183796f" (UID: "dd184b6f-3901-413f-ad6a-c7271183796f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:50.171046 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:50.171022 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd184b6f-3901-413f-ad6a-c7271183796f-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:50.171144 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:50.171049 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd184b6f-3901-413f-ad6a-c7271183796f-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:50.171144 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:50.171065 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd184b6f-3901-413f-ad6a-c7271183796f-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:50.171144 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:50.171079 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd184b6f-3901-413f-ad6a-c7271183796f-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:50.171144 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:50.171094 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd184b6f-3901-413f-ad6a-c7271183796f-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:50.172307 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:50.172282 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd184b6f-3901-413f-ad6a-c7271183796f-istio-podinfo" pod "dd184b6f-3901-413f-ad6a-c7271183796f" (UID: "dd184b6f-3901-413f-ad6a-c7271183796f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:25:50.172390 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:50.172332 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd184b6f-3901-413f-ad6a-c7271183796f-istio-token" pod "dd184b6f-3901-413f-ad6a-c7271183796f" (UID: "dd184b6f-3901-413f-ad6a-c7271183796f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:25:50.172527 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:50.172504 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd184b6f-3901-413f-ad6a-c7271183796f-istio-envoy" pod "dd184b6f-3901-413f-ad6a-c7271183796f" (UID: "dd184b6f-3901-413f-ad6a-c7271183796f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:50.172627 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:50.172554 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd184b6f-3901-413f-ad6a-c7271183796f-kube-api-access-lp72b" pod "dd184b6f-3901-413f-ad6a-c7271183796f" (UID: "dd184b6f-3901-413f-ad6a-c7271183796f"). InnerVolumeSpecName "kube-api-access-lp72b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:25:50.272838 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:50.272801 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd184b6f-3901-413f-ad6a-c7271183796f-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:50.272838 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:50.272835 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd184b6f-3901-413f-ad6a-c7271183796f-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:50.272838 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:50.272845 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd184b6f-3901-413f-ad6a-c7271183796f-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:50.273112 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:50.272855 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp72b\" (UniqueName: \"kubernetes.io/projected/dd184b6f-3901-413f-ad6a-c7271183796f-kube-api-access-lp72b\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:50.770665 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:50.770633 5076 generic.go:356] "Generic (PLEG): container finished" podID="dd184b6f-3901-413f-ad6a-c7271183796f" containerID="892a4ab283b16a8a62d8b6ad337a6d06a446b89f94647fd4c89beedca60ff4fe" exitCode=0 Nov 02 23:25:50.770836 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:50.770684 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" Nov 02 23:25:50.770836 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:50.770717 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" event={"ID":"dd184b6f-3901-413f-ad6a-c7271183796f","Type":"ContainerDied","Data":"892a4ab283b16a8a62d8b6ad337a6d06a446b89f94647fd4c89beedca60ff4fe"} Nov 02 23:25:50.770836 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:50.770775 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729" event={"ID":"dd184b6f-3901-413f-ad6a-c7271183796f","Type":"ContainerDied","Data":"d0d3d35661461734718db94e2aeaf94befa9f0a3f94707c416f45ffe8b05c604"} Nov 02 23:25:50.770836 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:50.770803 5076 scope.go:122] "RemoveContainer" containerID="892a4ab283b16a8a62d8b6ad337a6d06a446b89f94647fd4c89beedca60ff4fe" Nov 02 23:25:50.781681 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:50.781664 5076 scope.go:122] "RemoveContainer" containerID="cd88cb2e172b9fbd8f8043805751be2d839b447e7726bcc933dbecb51fbb208e" Nov 02 23:25:50.790927 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:50.790877 5076 scope.go:122] "RemoveContainer" containerID="892a4ab283b16a8a62d8b6ad337a6d06a446b89f94647fd4c89beedca60ff4fe" Nov 02 23:25:50.791186 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:25:50.791167 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"892a4ab283b16a8a62d8b6ad337a6d06a446b89f94647fd4c89beedca60ff4fe\": container with ID starting with 892a4ab283b16a8a62d8b6ad337a6d06a446b89f94647fd4c89beedca60ff4fe not found: ID does not exist" containerID="892a4ab283b16a8a62d8b6ad337a6d06a446b89f94647fd4c89beedca60ff4fe" Nov 02 23:25:50.791237 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:50.791196 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"892a4ab283b16a8a62d8b6ad337a6d06a446b89f94647fd4c89beedca60ff4fe"} err="failed to get container status \"892a4ab283b16a8a62d8b6ad337a6d06a446b89f94647fd4c89beedca60ff4fe\": rpc error: code = NotFound desc = could not find container \"892a4ab283b16a8a62d8b6ad337a6d06a446b89f94647fd4c89beedca60ff4fe\": container with ID starting with 892a4ab283b16a8a62d8b6ad337a6d06a446b89f94647fd4c89beedca60ff4fe not found: ID does not exist" Nov 02 23:25:50.791237 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:50.791215 5076 scope.go:122] "RemoveContainer" containerID="cd88cb2e172b9fbd8f8043805751be2d839b447e7726bcc933dbecb51fbb208e" Nov 02 23:25:50.791433 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:25:50.791417 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd88cb2e172b9fbd8f8043805751be2d839b447e7726bcc933dbecb51fbb208e\": container with ID starting with cd88cb2e172b9fbd8f8043805751be2d839b447e7726bcc933dbecb51fbb208e not found: ID does not exist" containerID="cd88cb2e172b9fbd8f8043805751be2d839b447e7726bcc933dbecb51fbb208e" Nov 02 23:25:50.791473 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:50.791439 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd88cb2e172b9fbd8f8043805751be2d839b447e7726bcc933dbecb51fbb208e"} err="failed to get container status \"cd88cb2e172b9fbd8f8043805751be2d839b447e7726bcc933dbecb51fbb208e\": rpc error: code = NotFound desc = could not find container \"cd88cb2e172b9fbd8f8043805751be2d839b447e7726bcc933dbecb51fbb208e\": container with ID starting with cd88cb2e172b9fbd8f8043805751be2d839b447e7726bcc933dbecb51fbb208e not found: ID does not exist" Nov 02 23:25:50.974192 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:50.974155 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729"] Nov 02 23:25:51.021933 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:51.021827 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a69f65-openshift-default-77fcc9b9cc-rf729"] Nov 02 23:25:51.150471 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:25:51.150440 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd184b6f-3901-413f-ad6a-c7271183796f" path="/var/lib/kubelet/pods/dd184b6f-3901-413f-ad6a-c7271183796f/volumes" Nov 02 23:26:02.893237 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:02.893203 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97"] Nov 02 23:26:02.893766 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:02.893752 5076 cpu_manager.go:414] "containerMap: removing container" podUID="c7bcd988-c4a5-46e6-bba1-c4346f21922d" containerName="istio-proxy" Nov 02 23:26:02.893814 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:02.893767 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7bcd988-c4a5-46e6-bba1-c4346f21922d" containerName="istio-proxy" Nov 02 23:26:02.893814 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:02.893789 5076 cpu_manager.go:414] "containerMap: removing container" podUID="3df93a02-4d85-4282-891a-fbb18a0e7e33" containerName="payload-pre-processing" Nov 02 23:26:02.893814 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:02.893799 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3df93a02-4d85-4282-891a-fbb18a0e7e33" containerName="payload-pre-processing" Nov 02 23:26:02.893940 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:02.893832 5076 cpu_manager.go:414] "containerMap: removing container" podUID="dd184b6f-3901-413f-ad6a-c7271183796f" containerName="istio-proxy" Nov 02 23:26:02.893940 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:02.893837 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd184b6f-3901-413f-ad6a-c7271183796f" containerName="istio-proxy" Nov 02 23:26:02.893940 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:02.893845 5076 cpu_manager.go:414] "containerMap: removing container" podUID="dd184b6f-3901-413f-ad6a-c7271183796f" containerName="istio-proxy" Nov 02 23:26:02.893940 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:02.893850 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd184b6f-3901-413f-ad6a-c7271183796f" containerName="istio-proxy" Nov 02 23:26:02.893940 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:02.893859 5076 cpu_manager.go:414] "containerMap: removing container" podUID="428ec111-bd83-465d-b67e-e3ffcfdc946c" containerName="payload-processing" Nov 02 23:26:02.893940 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:02.893864 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="428ec111-bd83-465d-b67e-e3ffcfdc946c" containerName="payload-processing" Nov 02 23:26:02.893940 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:02.893879 5076 cpu_manager.go:414] "containerMap: removing container" podUID="11c1048d-56a0-4abf-97a0-f0b99ed82d7c" containerName="maas-api" Nov 02 23:26:02.893940 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:02.893884 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11c1048d-56a0-4abf-97a0-f0b99ed82d7c" containerName="maas-api" Nov 02 23:26:02.894188 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:02.893992 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd184b6f-3901-413f-ad6a-c7271183796f" containerName="istio-proxy" Nov 02 23:26:02.894188 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:02.894002 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7bcd988-c4a5-46e6-bba1-c4346f21922d" containerName="istio-proxy" Nov 02 23:26:02.894188 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:02.894009 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd184b6f-3901-413f-ad6a-c7271183796f" containerName="istio-proxy" Nov 02 23:26:02.894188 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:02.894016 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="428ec111-bd83-465d-b67e-e3ffcfdc946c" containerName="payload-processing" Nov 02 23:26:02.894188 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:02.894023 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="11c1048d-56a0-4abf-97a0-f0b99ed82d7c" containerName="maas-api" Nov 02 23:26:02.894188 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:02.894028 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7bcd988-c4a5-46e6-bba1-c4346f21922d" containerName="istio-proxy" Nov 02 23:26:02.894188 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:02.894035 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="3df93a02-4d85-4282-891a-fbb18a0e7e33" containerName="payload-pre-processing" Nov 02 23:26:02.897533 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:02.897514 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97" Nov 02 23:26:02.947557 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:02.947522 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-443fdf-serving-cert\"" Nov 02 23:26:02.997401 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:02.997366 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97"] Nov 02 23:26:03.027252 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:03.027216 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8-openshift-service-ca\") pod \"maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97\" (UID: \"21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97" Nov 02 23:26:03.027453 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:03.027291 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8-metrics-certs\") pod \"maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97\" (UID: \"21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97" Nov 02 23:26:03.027453 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:03.027369 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8-maas-api-tls\") pod \"maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97\" (UID: \"21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97" Nov 02 23:26:03.027453 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:03.027399 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx6nb\" (UniqueName: \"kubernetes.io/projected/21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8-kube-api-access-qx6nb\") pod \"maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97\" (UID: \"21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97" Nov 02 23:26:03.129030 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:03.128989 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8-openshift-service-ca\") pod \"maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97\" (UID: \"21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97" Nov 02 23:26:03.129217 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:03.129058 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8-metrics-certs\") pod \"maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97\" (UID: \"21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97" Nov 02 23:26:03.129217 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:03.129099 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8-maas-api-tls\") pod \"maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97\" (UID: \"21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97" Nov 02 23:26:03.129217 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:03.129120 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx6nb\" (UniqueName: \"kubernetes.io/projected/21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8-kube-api-access-qx6nb\") pod \"maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97\" (UID: \"21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97" Nov 02 23:26:03.129325 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:26:03.129220 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-443fdf-serving-cert: secret "maas-api-e2e-api-b-443fdf-serving-cert" not found Nov 02 23:26:03.129325 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:26:03.129295 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8-maas-api-tls podName:21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8 nodeName:}" failed. No retries permitted until 2026-11-02 23:26:03.6292754 +0000 UTC m=+2012.597807177 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8-maas-api-tls") pod "maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97" (UID: "21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8") : secret "maas-api-e2e-api-b-443fdf-serving-cert" not found Nov 02 23:26:03.129687 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:03.129665 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8-openshift-service-ca\") pod \"maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97\" (UID: \"21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97" Nov 02 23:26:03.131614 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:03.131593 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8-metrics-certs\") pod \"maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97\" (UID: \"21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97" Nov 02 23:26:03.189326 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:03.189292 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx6nb\" (UniqueName: \"kubernetes.io/projected/21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8-kube-api-access-qx6nb\") pod \"maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97\" (UID: \"21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97" Nov 02 23:26:03.636104 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:03.636016 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8-maas-api-tls\") pod \"maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97\" (UID: \"21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97" Nov 02 23:26:03.638392 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:03.638365 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8-maas-api-tls\") pod \"maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97\" (UID: \"21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97" Nov 02 23:26:03.811435 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:03.811401 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97" Nov 02 23:26:04.185158 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:04.185130 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97"] Nov 02 23:26:04.186939 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:26:04.186909 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ba1f08_6cec_4c24_8fa0_745fdf3f3ce8.slice/crio-88175317708cae229e792c3a7c8544c5767f0e5478ef318d3c612320eddc2d3a WatchSource:0}: Error finding container 88175317708cae229e792c3a7c8544c5767f0e5478ef318d3c612320eddc2d3a: Status 404 returned error can't find the container with id 88175317708cae229e792c3a7c8544c5767f0e5478ef318d3c612320eddc2d3a Nov 02 23:26:04.865423 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:04.865390 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97" event={"ID":"21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8","Type":"ContainerStarted","Data":"88175317708cae229e792c3a7c8544c5767f0e5478ef318d3c612320eddc2d3a"} Nov 02 23:26:05.873597 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:05.873559 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97" event={"ID":"21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8","Type":"ContainerStarted","Data":"855069e5aeea29b8df92a0d244ec3b1dbde9001621bff8637e2f5f958d88e312"} Nov 02 23:26:05.874039 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:05.873669 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97" Nov 02 23:26:06.001915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:05.994177 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97" podStartSLOduration=2.9730558990000002 podStartE2EDuration="3.994157851s" totalImagesPullingTime="1.021101952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:26:05.992543942 +0000 UTC m=+2014.961075778" watchObservedRunningTime="2026-11-02 23:26:05.994157851 +0000 UTC m=+2014.962689637" Nov 02 23:26:07.443331 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:07.443286 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5"] Nov 02 23:26:07.443716 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:07.443490 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5" podUID="d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc" containerName="maas-api" containerID="cri-o://bb1584d2e4a7ab4bb0430cb48642610e719da6bb7b584599bc70fbe2d43c16c9" gracePeriod=30 Nov 02 23:26:07.729573 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:07.729542 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5" Nov 02 23:26:07.893253 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:07.893215 5076 generic.go:356] "Generic (PLEG): container finished" podID="d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc" containerID="bb1584d2e4a7ab4bb0430cb48642610e719da6bb7b584599bc70fbe2d43c16c9" exitCode=0 Nov 02 23:26:07.893442 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:07.893290 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5" Nov 02 23:26:07.893531 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:07.893267 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5" event={"ID":"d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc","Type":"ContainerDied","Data":"bb1584d2e4a7ab4bb0430cb48642610e719da6bb7b584599bc70fbe2d43c16c9"} Nov 02 23:26:07.893605 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:07.893589 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5" event={"ID":"d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc","Type":"ContainerDied","Data":"3a07ba2e6914487c578383b9d1ec090652f39ec180903d442338f8848d6d7bcf"} Nov 02 23:26:07.893677 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:07.893646 5076 scope.go:122] "RemoveContainer" containerID="bb1584d2e4a7ab4bb0430cb48642610e719da6bb7b584599bc70fbe2d43c16c9" Nov 02 23:26:07.897680 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:07.897646 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc-maas-api-tls\") pod \"d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc\" (UID: \"d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc\") " Nov 02 23:26:07.897818 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:07.897692 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc-metrics-certs\") pod \"d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc\" (UID: \"d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc\") " Nov 02 23:26:07.897818 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:07.897765 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc-openshift-service-ca\") pod \"d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc\" (UID: \"d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc\") " Nov 02 23:26:07.897961 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:07.897921 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc-kube-api-access-fv4fp\" (UniqueName: \"kubernetes.io/projected/d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc-kube-api-access-fv4fp\") pod \"d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc\" (UID: \"d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc\") " Nov 02 23:26:07.898496 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:07.898468 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc-openshift-service-ca" pod "d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc" (UID: "d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:26:07.900023 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:07.900001 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc-maas-api-tls" pod "d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc" (UID: "d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:26:07.900514 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:07.900491 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc-metrics-certs" pod "d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc" (UID: "d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:26:07.901604 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:07.901582 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc-kube-api-access-fv4fp" pod "d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc" (UID: "d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc"). InnerVolumeSpecName "kube-api-access-fv4fp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:26:07.907183 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:07.907162 5076 scope.go:122] "RemoveContainer" containerID="bb1584d2e4a7ab4bb0430cb48642610e719da6bb7b584599bc70fbe2d43c16c9" Nov 02 23:26:07.907506 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:26:07.907487 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb1584d2e4a7ab4bb0430cb48642610e719da6bb7b584599bc70fbe2d43c16c9\": container with ID starting with bb1584d2e4a7ab4bb0430cb48642610e719da6bb7b584599bc70fbe2d43c16c9 not found: ID does not exist" containerID="bb1584d2e4a7ab4bb0430cb48642610e719da6bb7b584599bc70fbe2d43c16c9" Nov 02 23:26:07.907579 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:07.907532 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb1584d2e4a7ab4bb0430cb48642610e719da6bb7b584599bc70fbe2d43c16c9"} err="failed to get container status \"bb1584d2e4a7ab4bb0430cb48642610e719da6bb7b584599bc70fbe2d43c16c9\": rpc error: code = NotFound desc = could not find container \"bb1584d2e4a7ab4bb0430cb48642610e719da6bb7b584599bc70fbe2d43c16c9\": container with ID starting with bb1584d2e4a7ab4bb0430cb48642610e719da6bb7b584599bc70fbe2d43c16c9 not found: ID does not exist" Nov 02 23:26:07.999478 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:07.999392 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:26:07.999478 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:07.999426 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:26:07.999478 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:07.999436 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:26:07.999478 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:07.999445 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv4fp\" (UniqueName: \"kubernetes.io/projected/d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc-kube-api-access-fv4fp\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:26:08.319831 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:08.319742 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5"] Nov 02 23:26:08.333373 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:08.333343 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f5790-747597b594-vwdg5"] Nov 02 23:26:09.148306 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:09.148271 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc" path="/var/lib/kubelet/pods/d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc/volumes" Nov 02 23:26:09.884172 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:26:09.884131 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:26:11.886620 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:11.886537 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97" Nov 02 23:26:13.881750 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:13.881707 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps"] Nov 02 23:26:13.882274 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:13.882259 5076 cpu_manager.go:414] "containerMap: removing container" podUID="c7bcd988-c4a5-46e6-bba1-c4346f21922d" containerName="istio-proxy" Nov 02 23:26:13.882318 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:13.882274 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7bcd988-c4a5-46e6-bba1-c4346f21922d" containerName="istio-proxy" Nov 02 23:26:13.882318 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:13.882283 5076 cpu_manager.go:414] "containerMap: removing container" podUID="d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc" containerName="maas-api" Nov 02 23:26:13.882318 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:13.882289 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc" containerName="maas-api" Nov 02 23:26:13.882414 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:13.882388 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9c92d5e-e50a-4e17-bc02-2b16c1bdb9dc" containerName="maas-api" Nov 02 23:26:13.886868 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:13.886847 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps" Nov 02 23:26:13.976439 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:13.976403 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps\" (UID: \"b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps" Nov 02 23:26:13.976621 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:13.976532 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gsq6\" (UniqueName: \"kubernetes.io/projected/b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3-kube-api-access-9gsq6\") pod \"maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps\" (UID: \"b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps" Nov 02 23:26:13.976621 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:13.976564 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3-metrics-certs\") pod \"maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps\" (UID: \"b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps" Nov 02 23:26:13.976621 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:13.976608 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps\" (UID: \"b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps" Nov 02 23:26:13.984136 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:13.984096 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-7fdfca-serving-cert\"" Nov 02 23:26:13.986930 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:13.986902 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps"] Nov 02 23:26:14.078161 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:14.078124 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gsq6\" (UniqueName: \"kubernetes.io/projected/b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3-kube-api-access-9gsq6\") pod \"maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps\" (UID: \"b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps" Nov 02 23:26:14.078161 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:14.078165 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3-metrics-certs\") pod \"maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps\" (UID: \"b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps" Nov 02 23:26:14.078386 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:14.078186 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps\" (UID: \"b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps" Nov 02 23:26:14.078386 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:26:14.078332 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fdfca-serving-cert: secret "maas-api-e2e-ipp-b-7fdfca-serving-cert" not found Nov 02 23:26:14.078494 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:26:14.078413 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3-maas-api-tls podName:b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3 nodeName:}" failed. No retries permitted until 2026-11-02 23:26:14.578392479 +0000 UTC m=+2023.546924258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3-maas-api-tls") pod "maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps" (UID: "b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3") : secret "maas-api-e2e-ipp-b-7fdfca-serving-cert" not found Nov 02 23:26:14.078494 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:14.078337 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps\" (UID: \"b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps" Nov 02 23:26:14.079025 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:14.079003 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps\" (UID: \"b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps" Nov 02 23:26:14.080676 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:14.080656 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3-metrics-certs\") pod \"maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps\" (UID: \"b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps" Nov 02 23:26:14.133194 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:14.133118 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gsq6\" (UniqueName: \"kubernetes.io/projected/b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3-kube-api-access-9gsq6\") pod \"maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps\" (UID: \"b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps" Nov 02 23:26:14.585310 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:14.585267 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps\" (UID: \"b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps" Nov 02 23:26:14.587838 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:14.587807 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps\" (UID: \"b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps" Nov 02 23:26:14.800345 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:14.800314 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps" Nov 02 23:26:15.173289 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:15.173257 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps"] Nov 02 23:26:15.174415 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:26:15.174388 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb615d17d_c437_4fe9_8e69_3fdbcfa0a1b3.slice/crio-b374cdc937eaefa90a5b8c8a508c6524ec0b89248b9e2fc9515a69a0bd66bc75 WatchSource:0}: Error finding container b374cdc937eaefa90a5b8c8a508c6524ec0b89248b9e2fc9515a69a0bd66bc75: Status 404 returned error can't find the container with id b374cdc937eaefa90a5b8c8a508c6524ec0b89248b9e2fc9515a69a0bd66bc75 Nov 02 23:26:15.943294 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:15.943261 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps" event={"ID":"b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3","Type":"ContainerStarted","Data":"b374cdc937eaefa90a5b8c8a508c6524ec0b89248b9e2fc9515a69a0bd66bc75"} Nov 02 23:26:16.959362 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:16.959323 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps" event={"ID":"b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3","Type":"ContainerStarted","Data":"c28e40e0f1feefa98b64ec8581c52360f57c4732603a6cf596306a9e1cccba6f"} Nov 02 23:26:16.959865 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:16.959470 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps" Nov 02 23:26:17.070423 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:17.070364 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps" podStartSLOduration=3.166346354 podStartE2EDuration="4.07035118s" totalImagesPullingTime="904.004826ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:26:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:26:17.068360617 +0000 UTC m=+2026.036892404" watchObservedRunningTime="2026-11-02 23:26:17.07035118 +0000 UTC m=+2026.038882963" Nov 02 23:26:18.211205 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:18.211167 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp"] Nov 02 23:26:18.215929 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:18.215905 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" Nov 02 23:26:18.356957 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:18.355656 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-3bf62f-openshift-default-dockercfg-755bw\"" Nov 02 23:26:18.356957 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:18.356586 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-workload-socket\") pod \"e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp\" (UID: \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\") " pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" Nov 02 23:26:18.356957 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:18.356631 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-workload-certs\") pod \"e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp\" (UID: \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\") " pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" Nov 02 23:26:18.356957 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:18.356655 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-istio-podinfo\") pod \"e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp\" (UID: \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\") " pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" Nov 02 23:26:18.356957 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:18.356683 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp9m4\" (UniqueName: \"kubernetes.io/projected/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-kube-api-access-sp9m4\") pod \"e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp\" (UID: \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\") " pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" Nov 02 23:26:18.356957 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:18.356752 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-credential-socket\") pod \"e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp\" (UID: \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\") " pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" Nov 02 23:26:18.356957 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:18.356837 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-istio-token\") pod \"e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp\" (UID: \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\") " pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" Nov 02 23:26:18.356957 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:18.356863 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-istio-data\") pod \"e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp\" (UID: \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\") " pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" Nov 02 23:26:18.356957 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:18.356939 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-istiod-ca-cert\") pod \"e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp\" (UID: \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\") " pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" Nov 02 23:26:18.357594 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:18.356998 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-istio-envoy\") pod \"e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp\" (UID: \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\") " pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" Nov 02 23:26:18.368314 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:18.360441 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp"] Nov 02 23:26:18.457943 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:18.457913 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-credential-socket\") pod \"e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp\" (UID: \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\") " pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" Nov 02 23:26:18.458115 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:18.458064 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-istio-token\") pod \"e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp\" (UID: \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\") " pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" Nov 02 23:26:18.458115 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:18.458087 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-istio-data\") pod \"e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp\" (UID: \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\") " pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" Nov 02 23:26:18.458193 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:18.458116 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-istiod-ca-cert\") pod \"e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp\" (UID: \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\") " pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" Nov 02 23:26:18.458193 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:18.458156 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-istio-envoy\") pod \"e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp\" (UID: \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\") " pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" Nov 02 23:26:18.458263 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:18.458250 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-workload-socket\") pod \"e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp\" (UID: \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\") " pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" Nov 02 23:26:18.458299 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:18.458277 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-workload-certs\") pod \"e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp\" (UID: \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\") " pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" Nov 02 23:26:18.458347 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:18.458302 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-istio-podinfo\") pod \"e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp\" (UID: \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\") " pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" Nov 02 23:26:18.458347 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:18.458332 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp9m4\" (UniqueName: \"kubernetes.io/projected/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-kube-api-access-sp9m4\") pod \"e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp\" (UID: \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\") " pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" Nov 02 23:26:18.458503 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:18.458355 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-credential-socket\") pod \"e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp\" (UID: \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\") " pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" Nov 02 23:26:18.458503 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:18.458434 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-istio-data\") pod \"e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp\" (UID: \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\") " pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" Nov 02 23:26:18.458684 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:18.458663 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-workload-socket\") pod \"e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp\" (UID: \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\") " pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" Nov 02 23:26:18.458782 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:18.458765 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-workload-certs\") pod \"e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp\" (UID: \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\") " pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" Nov 02 23:26:18.458903 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:18.458871 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-istiod-ca-cert\") pod \"e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp\" (UID: \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\") " pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" Nov 02 23:26:18.460590 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:18.460566 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-istio-envoy\") pod \"e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp\" (UID: \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\") " pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" Nov 02 23:26:18.461118 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:18.461097 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-istio-podinfo\") pod \"e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp\" (UID: \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\") " pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" Nov 02 23:26:18.512930 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:18.512818 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-istio-token\") pod \"e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp\" (UID: \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\") " pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" Nov 02 23:26:18.512930 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:18.512875 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp9m4\" (UniqueName: \"kubernetes.io/projected/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-kube-api-access-sp9m4\") pod \"e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp\" (UID: \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\") " pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" Nov 02 23:26:18.529804 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:18.529778 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" Nov 02 23:26:18.949885 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:18.949861 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp"] Nov 02 23:26:18.950648 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:26:18.950619 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fa02b25_4e1a_4c98_bfaa_8fa3b0372f91.slice/crio-51a50052ee7c65a7a187510ad51446d165d4a95d84cc2ac017cbf697bd5e2cc4 WatchSource:0}: Error finding container 51a50052ee7c65a7a187510ad51446d165d4a95d84cc2ac017cbf697bd5e2cc4: Status 404 returned error can't find the container with id 51a50052ee7c65a7a187510ad51446d165d4a95d84cc2ac017cbf697bd5e2cc4 Nov 02 23:26:18.953360 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:18.953329 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:26:18.953427 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:18.953399 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:26:18.953465 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:18.953427 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:26:18.972435 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:18.972403 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" event={"ID":"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91","Type":"ContainerStarted","Data":"51a50052ee7c65a7a187510ad51446d165d4a95d84cc2ac017cbf697bd5e2cc4"} Nov 02 23:26:19.980059 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:19.980017 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" event={"ID":"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91","Type":"ContainerStarted","Data":"e5d4bf19db2a1293fb208a5ebda6a1e629e4289d83626d9b996a3e7fe2fbb601"} Nov 02 23:26:20.176915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:20.164411 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" podStartSLOduration=2.164392331 podStartE2EDuration="2.164392331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:26:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:26:20.145377771 +0000 UTC m=+2029.113909582" watchObservedRunningTime="2026-11-02 23:26:20.164392331 +0000 UTC m=+2029.132924116" Nov 02 23:26:20.530219 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:20.530178 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" Nov 02 23:26:20.536113 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:20.536087 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" Nov 02 23:26:20.986320 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:20.986284 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" Nov 02 23:26:20.987224 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:20.987205 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" Nov 02 23:26:22.970390 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:22.970361 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps" Nov 02 23:26:31.372856 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:31.372774 5076 patch_prober.go:29] interesting pod/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 02 23:26:31.372856 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:31.372825 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" podUID="8c5807a1-409d-4188-abb7-1bf3bfb8e07d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 02 23:26:32.080197 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:32.080160 5076 generic.go:356] "Generic (PLEG): container finished" podID="8c5807a1-409d-4188-abb7-1bf3bfb8e07d" containerID="bb3b65d54e0566441a9ab3fadbc4e9cd77ea6305086388174b1fd87a19c4fdf6" exitCode=0 Nov 02 23:26:32.080364 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:32.080234 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" event={"ID":"8c5807a1-409d-4188-abb7-1bf3bfb8e07d","Type":"ContainerDied","Data":"bb3b65d54e0566441a9ab3fadbc4e9cd77ea6305086388174b1fd87a19c4fdf6"} Nov 02 23:26:32.080628 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:32.080614 5076 scope.go:122] "RemoveContainer" containerID="bb3b65d54e0566441a9ab3fadbc4e9cd77ea6305086388174b1fd87a19c4fdf6" Nov 02 23:26:32.082049 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:32.082023 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:26:32.082118 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:32.082088 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:26:32.082163 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:32.082121 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:26:33.088418 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:33.088379 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" event={"ID":"8c5807a1-409d-4188-abb7-1bf3bfb8e07d","Type":"ContainerStarted","Data":"c7800ec2dfb7b4ff4805a5b9a9a863134b456ee5f343daaea2c49ab10487e1e3"} Nov 02 23:26:33.395088 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:33.395002 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" Nov 02 23:26:33.395286 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:33.395262 5076 patch_prober.go:29] interesting pod/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 02 23:26:33.395333 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:33.395312 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" podUID="8c5807a1-409d-4188-abb7-1bf3bfb8e07d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 02 23:26:34.395146 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:34.395109 5076 patch_prober.go:29] interesting pod/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 02 23:26:34.395558 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:34.395162 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" podUID="8c5807a1-409d-4188-abb7-1bf3bfb8e07d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 02 23:26:35.394686 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:35.394648 5076 patch_prober.go:29] interesting pod/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 02 23:26:35.394868 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:35.394717 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" podUID="8c5807a1-409d-4188-abb7-1bf3bfb8e07d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 02 23:26:36.395422 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:36.395388 5076 patch_prober.go:29] interesting pod/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 02 23:26:36.395819 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:36.395441 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" podUID="8c5807a1-409d-4188-abb7-1bf3bfb8e07d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 02 23:26:36.537008 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:36.536969 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5bf989ccc5-mgxq9"] Nov 02 23:26:36.540093 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:36.540070 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5bf989ccc5-mgxq9" Nov 02 23:26:36.636552 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:36.636518 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-981724\"" Nov 02 23:26:36.636789 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:36.636635 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-981724-dockercfg-6rxsr\"" Nov 02 23:26:36.641200 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:36.641170 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5bf989ccc5-mgxq9"] Nov 02 23:26:36.652155 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:36.652075 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdnvk\" (UniqueName: \"kubernetes.io/projected/fb5eb5ca-ea96-4366-b376-3abac4ce1d8a-kube-api-access-mdnvk\") pod \"payload-pre-processing-e2e-ipp-a-981724-5bf989ccc5-mgxq9\" (UID: \"fb5eb5ca-ea96-4366-b376-3abac4ce1d8a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5bf989ccc5-mgxq9" Nov 02 23:26:36.652285 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:36.652177 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb5eb5ca-ea96-4366-b376-3abac4ce1d8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-981724-5bf989ccc5-mgxq9\" (UID: \"fb5eb5ca-ea96-4366-b376-3abac4ce1d8a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5bf989ccc5-mgxq9" Nov 02 23:26:36.753817 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:36.753780 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb5eb5ca-ea96-4366-b376-3abac4ce1d8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-981724-5bf989ccc5-mgxq9\" (UID: \"fb5eb5ca-ea96-4366-b376-3abac4ce1d8a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5bf989ccc5-mgxq9" Nov 02 23:26:36.754022 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:36.753989 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdnvk\" (UniqueName: \"kubernetes.io/projected/fb5eb5ca-ea96-4366-b376-3abac4ce1d8a-kube-api-access-mdnvk\") pod \"payload-pre-processing-e2e-ipp-a-981724-5bf989ccc5-mgxq9\" (UID: \"fb5eb5ca-ea96-4366-b376-3abac4ce1d8a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5bf989ccc5-mgxq9" Nov 02 23:26:36.754541 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:36.754519 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb5eb5ca-ea96-4366-b376-3abac4ce1d8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-981724-5bf989ccc5-mgxq9\" (UID: \"fb5eb5ca-ea96-4366-b376-3abac4ce1d8a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5bf989ccc5-mgxq9" Nov 02 23:26:36.808020 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:36.807988 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdnvk\" (UniqueName: \"kubernetes.io/projected/fb5eb5ca-ea96-4366-b376-3abac4ce1d8a-kube-api-access-mdnvk\") pod \"payload-pre-processing-e2e-ipp-a-981724-5bf989ccc5-mgxq9\" (UID: \"fb5eb5ca-ea96-4366-b376-3abac4ce1d8a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5bf989ccc5-mgxq9" Nov 02 23:26:36.852863 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:36.852829 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5bf989ccc5-mgxq9" Nov 02 23:26:37.235864 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:37.235817 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-981724-d5985fb4-s88v5"] Nov 02 23:26:37.240247 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:37.240219 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5bf989ccc5-mgxq9"] Nov 02 23:26:37.240382 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:37.240319 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-981724-d5985fb4-s88v5" Nov 02 23:26:37.274701 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:26:37.274675 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb5eb5ca_ea96_4366_b376_3abac4ce1d8a.slice/crio-9b0468e8f990a85da95a0deb2ad46d9f6e5b07a9b600a34f41ef3287fac70c98 WatchSource:0}: Error finding container 9b0468e8f990a85da95a0deb2ad46d9f6e5b07a9b600a34f41ef3287fac70c98: Status 404 returned error can't find the container with id 9b0468e8f990a85da95a0deb2ad46d9f6e5b07a9b600a34f41ef3287fac70c98 Nov 02 23:26:37.288351 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:37.288322 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-981724-d5985fb4-s88v5"] Nov 02 23:26:37.360421 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:37.360390 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5a2fe82-afb7-4822-b714-d44da311b6ed-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-981724-d5985fb4-s88v5\" (UID: \"d5a2fe82-afb7-4822-b714-d44da311b6ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-981724-d5985fb4-s88v5" Nov 02 23:26:37.360578 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:37.360438 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhv9v\" (UniqueName: \"kubernetes.io/projected/d5a2fe82-afb7-4822-b714-d44da311b6ed-kube-api-access-vhv9v\") pod \"payload-processing-e2e-ipp-a-981724-d5985fb4-s88v5\" (UID: \"d5a2fe82-afb7-4822-b714-d44da311b6ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-981724-d5985fb4-s88v5" Nov 02 23:26:37.394547 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:37.394515 5076 patch_prober.go:29] interesting pod/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 02 23:26:37.394704 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:37.394566 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" podUID="8c5807a1-409d-4188-abb7-1bf3bfb8e07d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 02 23:26:37.461491 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:37.461454 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5a2fe82-afb7-4822-b714-d44da311b6ed-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-981724-d5985fb4-s88v5\" (UID: \"d5a2fe82-afb7-4822-b714-d44da311b6ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-981724-d5985fb4-s88v5" Nov 02 23:26:37.461999 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:37.461501 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhv9v\" (UniqueName: \"kubernetes.io/projected/d5a2fe82-afb7-4822-b714-d44da311b6ed-kube-api-access-vhv9v\") pod \"payload-processing-e2e-ipp-a-981724-d5985fb4-s88v5\" (UID: \"d5a2fe82-afb7-4822-b714-d44da311b6ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-981724-d5985fb4-s88v5" Nov 02 23:26:37.462434 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:37.462405 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5a2fe82-afb7-4822-b714-d44da311b6ed-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-981724-d5985fb4-s88v5\" (UID: \"d5a2fe82-afb7-4822-b714-d44da311b6ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-981724-d5985fb4-s88v5" Nov 02 23:26:37.536012 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:37.535881 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhv9v\" (UniqueName: \"kubernetes.io/projected/d5a2fe82-afb7-4822-b714-d44da311b6ed-kube-api-access-vhv9v\") pod \"payload-processing-e2e-ipp-a-981724-d5985fb4-s88v5\" (UID: \"d5a2fe82-afb7-4822-b714-d44da311b6ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-981724-d5985fb4-s88v5" Nov 02 23:26:37.563298 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:37.563260 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-981724-d5985fb4-s88v5" Nov 02 23:26:37.957732 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:37.957700 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-981724-d5985fb4-s88v5"] Nov 02 23:26:37.958916 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:26:37.958854 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a2fe82_afb7_4822_b714_d44da311b6ed.slice/crio-e6ce95a450bedc0d356a8f271118a4d91e7ec093a28b29205ba6e3d405f4f98c WatchSource:0}: Error finding container e6ce95a450bedc0d356a8f271118a4d91e7ec093a28b29205ba6e3d405f4f98c: Status 404 returned error can't find the container with id e6ce95a450bedc0d356a8f271118a4d91e7ec093a28b29205ba6e3d405f4f98c Nov 02 23:26:37.989443 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:37.989414 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm"] Nov 02 23:26:37.993044 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:37.993027 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm" Nov 02 23:26:38.042771 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:38.042744 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-81c3c9-serving-cert\"" Nov 02 23:26:38.062644 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:38.062528 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm"] Nov 02 23:26:38.069227 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:38.069188 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/145c9784-8db4-4686-b77d-7f33d51272d0-maas-api-tls\") pod \"maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm\" (UID: \"145c9784-8db4-4686-b77d-7f33d51272d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm" Nov 02 23:26:38.069664 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:38.069628 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwqfx\" (UniqueName: \"kubernetes.io/projected/145c9784-8db4-4686-b77d-7f33d51272d0-kube-api-access-hwqfx\") pod \"maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm\" (UID: \"145c9784-8db4-4686-b77d-7f33d51272d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm" Nov 02 23:26:38.069783 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:38.069747 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/145c9784-8db4-4686-b77d-7f33d51272d0-metrics-certs\") pod \"maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm\" (UID: \"145c9784-8db4-4686-b77d-7f33d51272d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm" Nov 02 23:26:38.069872 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:38.069852 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/145c9784-8db4-4686-b77d-7f33d51272d0-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm\" (UID: \"145c9784-8db4-4686-b77d-7f33d51272d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm" Nov 02 23:26:38.145380 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:38.145341 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5bf989ccc5-mgxq9" event={"ID":"fb5eb5ca-ea96-4366-b376-3abac4ce1d8a","Type":"ContainerStarted","Data":"263d8d00024e4886c14e6a4560721270211610b4fedb6619fdde5209378a6056"} Nov 02 23:26:38.145563 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:38.145400 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5bf989ccc5-mgxq9" event={"ID":"fb5eb5ca-ea96-4366-b376-3abac4ce1d8a","Type":"ContainerStarted","Data":"9b0468e8f990a85da95a0deb2ad46d9f6e5b07a9b600a34f41ef3287fac70c98"} Nov 02 23:26:38.145563 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:38.145470 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5bf989ccc5-mgxq9" Nov 02 23:26:38.147850 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:38.147828 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-981724-d5985fb4-s88v5" event={"ID":"d5a2fe82-afb7-4822-b714-d44da311b6ed","Type":"ContainerStarted","Data":"501f4b1a7943ed29522d56ae1682af3b4b9c3e13938461db3e682e82ad03c260"} Nov 02 23:26:38.147984 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:38.147863 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-981724-d5985fb4-s88v5" event={"ID":"d5a2fe82-afb7-4822-b714-d44da311b6ed","Type":"ContainerStarted","Data":"e6ce95a450bedc0d356a8f271118a4d91e7ec093a28b29205ba6e3d405f4f98c"} Nov 02 23:26:38.170809 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:38.170776 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/145c9784-8db4-4686-b77d-7f33d51272d0-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm\" (UID: \"145c9784-8db4-4686-b77d-7f33d51272d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm" Nov 02 23:26:38.170964 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:38.170817 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/145c9784-8db4-4686-b77d-7f33d51272d0-maas-api-tls\") pod \"maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm\" (UID: \"145c9784-8db4-4686-b77d-7f33d51272d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm" Nov 02 23:26:38.171038 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:38.170966 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwqfx\" (UniqueName: \"kubernetes.io/projected/145c9784-8db4-4686-b77d-7f33d51272d0-kube-api-access-hwqfx\") pod \"maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm\" (UID: \"145c9784-8db4-4686-b77d-7f33d51272d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm" Nov 02 23:26:38.171038 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:26:38.171011 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-81c3c9-serving-cert: secret "maas-api-e2e-praxis-a-81c3c9-serving-cert" not found Nov 02 23:26:38.171150 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:38.171063 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/145c9784-8db4-4686-b77d-7f33d51272d0-metrics-certs\") pod \"maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm\" (UID: \"145c9784-8db4-4686-b77d-7f33d51272d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm" Nov 02 23:26:38.171150 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:26:38.171080 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/145c9784-8db4-4686-b77d-7f33d51272d0-maas-api-tls podName:145c9784-8db4-4686-b77d-7f33d51272d0 nodeName:}" failed. No retries permitted until 2026-11-02 23:26:38.671059772 +0000 UTC m=+2047.639591559 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/145c9784-8db4-4686-b77d-7f33d51272d0-maas-api-tls") pod "maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm" (UID: "145c9784-8db4-4686-b77d-7f33d51272d0") : secret "maas-api-e2e-praxis-a-81c3c9-serving-cert" not found Nov 02 23:26:38.171641 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:38.171622 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/145c9784-8db4-4686-b77d-7f33d51272d0-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm\" (UID: \"145c9784-8db4-4686-b77d-7f33d51272d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm" Nov 02 23:26:38.173466 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:38.173442 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/145c9784-8db4-4686-b77d-7f33d51272d0-metrics-certs\") pod \"maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm\" (UID: \"145c9784-8db4-4686-b77d-7f33d51272d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm" Nov 02 23:26:38.228995 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:38.226552 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwqfx\" (UniqueName: \"kubernetes.io/projected/145c9784-8db4-4686-b77d-7f33d51272d0-kube-api-access-hwqfx\") pod \"maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm\" (UID: \"145c9784-8db4-4686-b77d-7f33d51272d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm" Nov 02 23:26:38.364787 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:38.364727 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-981724-d5985fb4-s88v5" podStartSLOduration=1.364713381 podStartE2EDuration="1.364713381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:26:38.361926424 +0000 UTC m=+2047.330458210" watchObservedRunningTime="2026-11-02 23:26:38.364713381 +0000 UTC m=+2047.333245165" Nov 02 23:26:38.365195 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:38.365166 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5bf989ccc5-mgxq9" podStartSLOduration=2.365154434 podStartE2EDuration="2.365154434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:26:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:26:38.254807956 +0000 UTC m=+2047.223339739" watchObservedRunningTime="2026-11-02 23:26:38.365154434 +0000 UTC m=+2047.333686218" Nov 02 23:26:38.400133 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:38.400102 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" Nov 02 23:26:38.400536 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:38.400515 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" Nov 02 23:26:38.401265 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:38.401245 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" Nov 02 23:26:38.680260 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:38.680165 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/145c9784-8db4-4686-b77d-7f33d51272d0-maas-api-tls\") pod \"maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm\" (UID: \"145c9784-8db4-4686-b77d-7f33d51272d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm" Nov 02 23:26:38.682801 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:38.682770 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/145c9784-8db4-4686-b77d-7f33d51272d0-maas-api-tls\") pod \"maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm\" (UID: \"145c9784-8db4-4686-b77d-7f33d51272d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm" Nov 02 23:26:38.933206 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:38.933122 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm" Nov 02 23:26:39.154858 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:39.154826 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-981724-d5985fb4-s88v5" Nov 02 23:26:39.317107 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:26:39.317078 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod145c9784_8db4_4686_b77d_7f33d51272d0.slice/crio-a960e7f0235073f1af02e0ac7256e13b4ef7853f2d9a8ced941c30c72f753384 WatchSource:0}: Error finding container a960e7f0235073f1af02e0ac7256e13b4ef7853f2d9a8ced941c30c72f753384: Status 404 returned error can't find the container with id a960e7f0235073f1af02e0ac7256e13b4ef7853f2d9a8ced941c30c72f753384 Nov 02 23:26:39.317441 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:39.317414 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm"] Nov 02 23:26:39.884934 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:26:39.884872 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:26:40.161949 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:40.161919 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm" event={"ID":"145c9784-8db4-4686-b77d-7f33d51272d0","Type":"ContainerStarted","Data":"a960e7f0235073f1af02e0ac7256e13b4ef7853f2d9a8ced941c30c72f753384"} Nov 02 23:26:41.169013 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:41.168978 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm" event={"ID":"145c9784-8db4-4686-b77d-7f33d51272d0","Type":"ContainerStarted","Data":"258ca72e34619731e50617d34660013bad571136e81dbb8147bf5b775afb6bda"} Nov 02 23:26:41.169516 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:41.169100 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm" Nov 02 23:26:41.290539 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:41.287249 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm" podStartSLOduration=3.494707149 podStartE2EDuration="4.287225725s" totalImagesPullingTime="792.518576ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:26:41.282666647 +0000 UTC m=+2050.251198429" watchObservedRunningTime="2026-11-02 23:26:41.287225725 +0000 UTC m=+2050.255757513" Nov 02 23:26:47.180764 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:47.180735 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm" Nov 02 23:26:49.157271 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:49.157240 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5bf989ccc5-mgxq9" Nov 02 23:26:50.164383 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:50.164352 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-981724-d5985fb4-s88v5" Nov 02 23:26:53.659496 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:53.659459 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv"] Nov 02 23:26:53.662529 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:53.662504 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" Nov 02 23:26:53.739019 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:53.738983 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-istio-envoy\") pod \"e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv\" (UID: \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\") " pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" Nov 02 23:26:53.739311 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:53.739275 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-credential-socket\") pod \"e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv\" (UID: \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\") " pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" Nov 02 23:26:53.739846 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:53.739809 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-istiod-ca-cert\") pod \"e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv\" (UID: \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\") " pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" Nov 02 23:26:53.740064 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:53.739927 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-istio-data\") pod \"e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv\" (UID: \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\") " pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" Nov 02 23:26:53.740064 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:53.739971 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpvzk\" (UniqueName: \"kubernetes.io/projected/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-kube-api-access-vpvzk\") pod \"e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv\" (UID: \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\") " pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" Nov 02 23:26:53.740064 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:53.740039 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-istio-podinfo\") pod \"e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv\" (UID: \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\") " pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" Nov 02 23:26:53.740239 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:53.740068 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-workload-certs\") pod \"e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv\" (UID: \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\") " pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" Nov 02 23:26:53.740239 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:53.740130 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-istio-token\") pod \"e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv\" (UID: \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\") " pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" Nov 02 23:26:53.740239 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:53.740157 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-workload-socket\") pod \"e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv\" (UID: \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\") " pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" Nov 02 23:26:53.740484 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:53.739732 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-5ff336-openshift-default-dockercfg-chsmv\"" Nov 02 23:26:53.779467 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:53.779427 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv"] Nov 02 23:26:53.848041 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:53.841403 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-istio-envoy\") pod \"e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv\" (UID: \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\") " pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" Nov 02 23:26:53.848041 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:53.841462 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-credential-socket\") pod \"e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv\" (UID: \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\") " pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" Nov 02 23:26:53.848041 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:53.841519 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-istiod-ca-cert\") pod \"e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv\" (UID: \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\") " pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" Nov 02 23:26:53.848041 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:53.841600 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-istio-data\") pod \"e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv\" (UID: \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\") " pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" Nov 02 23:26:53.848041 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:53.841645 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpvzk\" (UniqueName: \"kubernetes.io/projected/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-kube-api-access-vpvzk\") pod \"e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv\" (UID: \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\") " pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" Nov 02 23:26:53.848041 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:53.841714 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-istio-podinfo\") pod \"e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv\" (UID: \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\") " pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" Nov 02 23:26:53.848041 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:53.841742 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-workload-certs\") pod \"e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv\" (UID: \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\") " pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" Nov 02 23:26:53.848041 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:53.841802 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-istio-token\") pod \"e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv\" (UID: \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\") " pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" Nov 02 23:26:53.848041 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:53.841829 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-workload-socket\") pod \"e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv\" (UID: \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\") " pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" Nov 02 23:26:53.848041 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:53.842498 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-workload-socket\") pod \"e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv\" (UID: \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\") " pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" Nov 02 23:26:53.856367 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:53.852504 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-credential-socket\") pod \"e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv\" (UID: \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\") " pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" Nov 02 23:26:53.856367 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:53.855318 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-istio-data\") pod \"e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv\" (UID: \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\") " pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" Nov 02 23:26:53.856367 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:53.855495 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-istio-envoy\") pod \"e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv\" (UID: \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\") " pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" Nov 02 23:26:53.856367 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:53.856192 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-workload-certs\") pod \"e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv\" (UID: \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\") " pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" Nov 02 23:26:53.879374 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:53.879283 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-istiod-ca-cert\") pod \"e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv\" (UID: \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\") " pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" Nov 02 23:26:53.893028 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:53.892988 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-istio-podinfo\") pod \"e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv\" (UID: \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\") " pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" Nov 02 23:26:53.907876 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:53.907836 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpvzk\" (UniqueName: \"kubernetes.io/projected/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-kube-api-access-vpvzk\") pod \"e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv\" (UID: \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\") " pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" Nov 02 23:26:53.912915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:53.912833 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-istio-token\") pod \"e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv\" (UID: \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\") " pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" Nov 02 23:26:53.976303 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:53.975998 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" Nov 02 23:26:54.398209 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:54.398178 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv"] Nov 02 23:26:54.399216 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:26:54.399184 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef0ec1d5_f3b2_4b9e_a86b_d2a158f997c9.slice/crio-d63e2b20e38dfbd3b9557a99a59b1468f14b8e4f4f116469d51ffa091bc888f3 WatchSource:0}: Error finding container d63e2b20e38dfbd3b9557a99a59b1468f14b8e4f4f116469d51ffa091bc888f3: Status 404 returned error can't find the container with id d63e2b20e38dfbd3b9557a99a59b1468f14b8e4f4f116469d51ffa091bc888f3 Nov 02 23:26:54.402644 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:54.402610 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:26:54.402766 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:54.402691 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:26:54.402766 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:54.402718 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:26:55.278403 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:55.278366 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" event={"ID":"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9","Type":"ContainerStarted","Data":"25d296194a17393f8f3e7d15b994b96d25cbb18c18540c4c94e0ecca5d3da43a"} Nov 02 23:26:55.278838 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:55.278427 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" event={"ID":"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9","Type":"ContainerStarted","Data":"d63e2b20e38dfbd3b9557a99a59b1468f14b8e4f4f116469d51ffa091bc888f3"} Nov 02 23:26:55.440496 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:55.440442 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" podStartSLOduration=2.440428246 podStartE2EDuration="2.440428246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:26:55.439064192 +0000 UTC m=+2064.407595979" watchObservedRunningTime="2026-11-02 23:26:55.440428246 +0000 UTC m=+2064.408960087" Nov 02 23:26:55.976602 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:55.976563 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" Nov 02 23:26:55.982870 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:55.982845 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" Nov 02 23:26:56.286724 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:56.286647 5076 generic.go:356] "Generic (PLEG): container finished" podID="c3d41320-5841-4706-9e82-e5619986d711" containerID="0ef99bfbdc0426efb15add5364e6a19faa32f368b40c8e6e1edef5523018c00e" exitCode=0 Nov 02 23:26:56.287141 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:56.286717 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" event={"ID":"c3d41320-5841-4706-9e82-e5619986d711","Type":"ContainerDied","Data":"0ef99bfbdc0426efb15add5364e6a19faa32f368b40c8e6e1edef5523018c00e"} Nov 02 23:26:56.287141 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:56.287117 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" Nov 02 23:26:56.287214 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:56.287165 5076 scope.go:122] "RemoveContainer" containerID="0ef99bfbdc0426efb15add5364e6a19faa32f368b40c8e6e1edef5523018c00e" Nov 02 23:26:56.288113 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:56.288099 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" Nov 02 23:26:56.288641 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:56.288613 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:26:56.288698 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:56.288684 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:26:56.288736 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:56.288718 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:26:57.295166 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:57.295126 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" event={"ID":"c3d41320-5841-4706-9e82-e5619986d711","Type":"ContainerStarted","Data":"3f5f1763c85a39a78f247fa13bde6c8cd5da438d76008537260d4b8c7ce94a13"} Nov 02 23:26:58.242472 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:58.242438 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" Nov 02 23:26:58.243815 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:58.243788 5076 patch_prober.go:29] interesting pod/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 02 23:26:58.243964 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:58.243835 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" podUID="c3d41320-5841-4706-9e82-e5619986d711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 02 23:26:59.242172 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:59.242134 5076 patch_prober.go:29] interesting pod/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 02 23:26:59.242603 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:26:59.242203 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" podUID="c3d41320-5841-4706-9e82-e5619986d711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 02 23:27:00.242414 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:27:00.242376 5076 patch_prober.go:29] interesting pod/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 02 23:27:00.242828 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:27:00.242428 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" podUID="c3d41320-5841-4706-9e82-e5619986d711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 02 23:27:01.242613 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:27:01.242573 5076 patch_prober.go:29] interesting pod/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 02 23:27:01.243056 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:27:01.242629 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" podUID="c3d41320-5841-4706-9e82-e5619986d711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 02 23:27:01.535165 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:27:01.535071 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" Nov 02 23:27:02.265610 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:27:02.265571 5076 patch_prober.go:29] interesting pod/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 02 23:27:02.266111 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:27:02.265635 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" podUID="c3d41320-5841-4706-9e82-e5619986d711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 02 23:27:03.246806 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:27:03.246772 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" Nov 02 23:27:03.247986 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:27:03.247965 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" Nov 02 23:27:09.884879 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:27:09.884844 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:27:14.424351 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:27:14.424316 5076 generic.go:356] "Generic (PLEG): container finished" podID="8bcab402-b870-4284-9d80-a7c8243492bd" containerID="8c6b87f5d8752b4cfb609abe74bb9468ddeaedd3545e3d93c816bf11b3814cc4" exitCode=0 Nov 02 23:27:14.424777 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:27:14.424393 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" event={"ID":"8bcab402-b870-4284-9d80-a7c8243492bd","Type":"ContainerDied","Data":"8c6b87f5d8752b4cfb609abe74bb9468ddeaedd3545e3d93c816bf11b3814cc4"} Nov 02 23:27:14.424878 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:27:14.424863 5076 scope.go:122] "RemoveContainer" containerID="8c6b87f5d8752b4cfb609abe74bb9468ddeaedd3545e3d93c816bf11b3814cc4" Nov 02 23:27:14.426471 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:27:14.426445 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:27:14.426548 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:27:14.426510 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:27:14.426548 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:27:14.426541 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:27:15.439713 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:27:15.439678 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" event={"ID":"8bcab402-b870-4284-9d80-a7c8243492bd","Type":"ContainerStarted","Data":"f742ef537c7215ea5da6f9eb2fd32e095b6cbc434e2704b06ef5bfea32dbf1c5"} Nov 02 23:27:15.888156 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:27:15.888069 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" Nov 02 23:27:15.889808 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:27:15.889770 5076 patch_prober.go:29] interesting pod/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 02 23:27:15.890020 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:27:15.889857 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" podUID="8bcab402-b870-4284-9d80-a7c8243492bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 02 23:27:16.887510 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:27:16.887472 5076 patch_prober.go:29] interesting pod/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 02 23:27:16.887932 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:27:16.887529 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" podUID="8bcab402-b870-4284-9d80-a7c8243492bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 02 23:27:17.887424 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:27:17.887387 5076 patch_prober.go:29] interesting pod/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 02 23:27:17.887657 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:27:17.887460 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" podUID="8bcab402-b870-4284-9d80-a7c8243492bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 02 23:27:18.653777 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:27:18.653734 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" Nov 02 23:27:18.886664 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:27:18.886629 5076 patch_prober.go:29] interesting pod/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 02 23:27:18.886841 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:27:18.886688 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" podUID="8bcab402-b870-4284-9d80-a7c8243492bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 02 23:27:19.887528 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:27:19.887498 5076 patch_prober.go:29] interesting pod/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 02 23:27:19.887948 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:27:19.887549 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" podUID="8bcab402-b870-4284-9d80-a7c8243492bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 02 23:27:20.891361 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:27:20.891326 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" Nov 02 23:27:20.892499 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:27:20.892477 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" Nov 02 23:27:31.756828 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:27:31.756797 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 02 23:27:31.798883 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:27:31.798848 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 02 23:27:39.883809 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:27:39.883773 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:28:09.554746 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:09.554660 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm"] Nov 02 23:28:09.555377 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:09.554881 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" podUID="dfd27984-010f-441c-a3f4-553933c0529a" containerName="istio-proxy" containerID="cri-o://db7259016578f9d9120ee7dc14f0d84c043ede86b683c463e84d7f64a4a5eb63" gracePeriod=30 Nov 02 23:28:09.886266 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:28:09.886179 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:28:10.036155 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:10.036121 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr"] Nov 02 23:28:10.036344 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:10.036324 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr" podUID="6ec4a99a-59d0-422c-98f4-066c5e3eb1a1" containerName="maas-api" containerID="cri-o://058d3ac6c51a8c272e3c09e095b0525a1df6e4e8e09ea97ac1ef61f429020737" gracePeriod=30 Nov 02 23:28:10.346076 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:10.346051 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr" Nov 02 23:28:10.511089 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:10.511049 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ec4a99a-59d0-422c-98f4-066c5e3eb1a1-kube-api-access-kh2k6\" (UniqueName: \"kubernetes.io/projected/6ec4a99a-59d0-422c-98f4-066c5e3eb1a1-kube-api-access-kh2k6\") pod \"6ec4a99a-59d0-422c-98f4-066c5e3eb1a1\" (UID: \"6ec4a99a-59d0-422c-98f4-066c5e3eb1a1\") " Nov 02 23:28:10.511292 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:10.511102 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ec4a99a-59d0-422c-98f4-066c5e3eb1a1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ec4a99a-59d0-422c-98f4-066c5e3eb1a1-metrics-certs\") pod \"6ec4a99a-59d0-422c-98f4-066c5e3eb1a1\" (UID: \"6ec4a99a-59d0-422c-98f4-066c5e3eb1a1\") " Nov 02 23:28:10.511292 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:10.511135 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ec4a99a-59d0-422c-98f4-066c5e3eb1a1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ec4a99a-59d0-422c-98f4-066c5e3eb1a1-maas-api-tls\") pod \"6ec4a99a-59d0-422c-98f4-066c5e3eb1a1\" (UID: \"6ec4a99a-59d0-422c-98f4-066c5e3eb1a1\") " Nov 02 23:28:10.511292 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:10.511249 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ec4a99a-59d0-422c-98f4-066c5e3eb1a1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ec4a99a-59d0-422c-98f4-066c5e3eb1a1-openshift-service-ca\") pod \"6ec4a99a-59d0-422c-98f4-066c5e3eb1a1\" (UID: \"6ec4a99a-59d0-422c-98f4-066c5e3eb1a1\") " Nov 02 23:28:10.511651 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:10.511623 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ec4a99a-59d0-422c-98f4-066c5e3eb1a1-openshift-service-ca" pod "6ec4a99a-59d0-422c-98f4-066c5e3eb1a1" (UID: "6ec4a99a-59d0-422c-98f4-066c5e3eb1a1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:28:10.511845 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:10.511827 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ec4a99a-59d0-422c-98f4-066c5e3eb1a1-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:10.513403 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:10.513373 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ec4a99a-59d0-422c-98f4-066c5e3eb1a1-metrics-certs" pod "6ec4a99a-59d0-422c-98f4-066c5e3eb1a1" (UID: "6ec4a99a-59d0-422c-98f4-066c5e3eb1a1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:28:10.513403 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:10.513392 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ec4a99a-59d0-422c-98f4-066c5e3eb1a1-maas-api-tls" pod "6ec4a99a-59d0-422c-98f4-066c5e3eb1a1" (UID: "6ec4a99a-59d0-422c-98f4-066c5e3eb1a1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:28:10.513526 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:10.513382 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ec4a99a-59d0-422c-98f4-066c5e3eb1a1-kube-api-access-kh2k6" pod "6ec4a99a-59d0-422c-98f4-066c5e3eb1a1" (UID: "6ec4a99a-59d0-422c-98f4-066c5e3eb1a1"). InnerVolumeSpecName "kube-api-access-kh2k6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:28:10.600197 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:10.600162 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6a894c-5dbb4748-58dmg"] Nov 02 23:28:10.600879 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:10.600831 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a894c-5dbb4748-58dmg" podUID="7542df71-b433-4496-b175-1797e098f9fe" containerName="payload-processing" containerID="cri-o://78d74cc7cfd646c8520d9a00f89449f4b8ec436e6ede934dba43ce7271a2fa14" gracePeriod=30 Nov 02 23:28:10.622568 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:10.622474 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh2k6\" (UniqueName: \"kubernetes.io/projected/6ec4a99a-59d0-422c-98f4-066c5e3eb1a1-kube-api-access-kh2k6\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:10.622568 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:10.622519 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ec4a99a-59d0-422c-98f4-066c5e3eb1a1-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:10.622568 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:10.622538 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ec4a99a-59d0-422c-98f4-066c5e3eb1a1-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:10.659264 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:10.659229 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-78b54bb9f9-rtscx"] Nov 02 23:28:10.659753 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:10.659705 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-78b54bb9f9-rtscx" podUID="0f98b219-1d3d-47d4-aaf1-3d06149307eb" containerName="payload-pre-processing" containerID="cri-o://92ce1fa84d3d751e6d06cd2640b0a5c19cfe359c5dc5755b0c6be09fa9e76f19" gracePeriod=30 Nov 02 23:28:10.851795 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:10.843453 5076 generic.go:356] "Generic (PLEG): container finished" podID="0f98b219-1d3d-47d4-aaf1-3d06149307eb" containerID="92ce1fa84d3d751e6d06cd2640b0a5c19cfe359c5dc5755b0c6be09fa9e76f19" exitCode=0 Nov 02 23:28:10.851795 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:10.843644 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-78b54bb9f9-rtscx" event={"ID":"0f98b219-1d3d-47d4-aaf1-3d06149307eb","Type":"ContainerDied","Data":"92ce1fa84d3d751e6d06cd2640b0a5c19cfe359c5dc5755b0c6be09fa9e76f19"} Nov 02 23:28:10.851795 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:10.845567 5076 generic.go:356] "Generic (PLEG): container finished" podID="6ec4a99a-59d0-422c-98f4-066c5e3eb1a1" containerID="058d3ac6c51a8c272e3c09e095b0525a1df6e4e8e09ea97ac1ef61f429020737" exitCode=0 Nov 02 23:28:10.851795 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:10.845689 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr" event={"ID":"6ec4a99a-59d0-422c-98f4-066c5e3eb1a1","Type":"ContainerDied","Data":"058d3ac6c51a8c272e3c09e095b0525a1df6e4e8e09ea97ac1ef61f429020737"} Nov 02 23:28:10.851795 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:10.845733 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr" event={"ID":"6ec4a99a-59d0-422c-98f4-066c5e3eb1a1","Type":"ContainerDied","Data":"4a3001cc99cda7571beb9a7ffca53fb9f118845e2864340df33074013544f53b"} Nov 02 23:28:10.851795 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:10.845762 5076 scope.go:122] "RemoveContainer" containerID="058d3ac6c51a8c272e3c09e095b0525a1df6e4e8e09ea97ac1ef61f429020737" Nov 02 23:28:10.851795 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:10.845924 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr" Nov 02 23:28:10.854597 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:10.854571 5076 generic.go:356] "Generic (PLEG): container finished" podID="7542df71-b433-4496-b175-1797e098f9fe" containerID="78d74cc7cfd646c8520d9a00f89449f4b8ec436e6ede934dba43ce7271a2fa14" exitCode=0 Nov 02 23:28:10.854838 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:10.854763 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a894c-5dbb4748-58dmg" event={"ID":"7542df71-b433-4496-b175-1797e098f9fe","Type":"ContainerDied","Data":"78d74cc7cfd646c8520d9a00f89449f4b8ec436e6ede934dba43ce7271a2fa14"} Nov 02 23:28:10.871209 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:10.871105 5076 scope.go:122] "RemoveContainer" containerID="058d3ac6c51a8c272e3c09e095b0525a1df6e4e8e09ea97ac1ef61f429020737" Nov 02 23:28:10.872117 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:28:10.872034 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"058d3ac6c51a8c272e3c09e095b0525a1df6e4e8e09ea97ac1ef61f429020737\": container with ID starting with 058d3ac6c51a8c272e3c09e095b0525a1df6e4e8e09ea97ac1ef61f429020737 not found: ID does not exist" containerID="058d3ac6c51a8c272e3c09e095b0525a1df6e4e8e09ea97ac1ef61f429020737" Nov 02 23:28:10.872117 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:10.872082 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"058d3ac6c51a8c272e3c09e095b0525a1df6e4e8e09ea97ac1ef61f429020737"} err="failed to get container status \"058d3ac6c51a8c272e3c09e095b0525a1df6e4e8e09ea97ac1ef61f429020737\": rpc error: code = NotFound desc = could not find container \"058d3ac6c51a8c272e3c09e095b0525a1df6e4e8e09ea97ac1ef61f429020737\": container with ID starting with 058d3ac6c51a8c272e3c09e095b0525a1df6e4e8e09ea97ac1ef61f429020737 not found: ID does not exist" Nov 02 23:28:10.951619 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:10.951585 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a894c-5dbb4748-58dmg" Nov 02 23:28:10.965776 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:10.965748 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-78b54bb9f9-rtscx" Nov 02 23:28:11.005047 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:11.004990 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr"] Nov 02 23:28:11.008202 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:11.008180 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6a894c-849f8dd5c6-2zfbr"] Nov 02 23:28:11.161911 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:11.160702 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f98b219-1d3d-47d4-aaf1-3d06149307eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f98b219-1d3d-47d4-aaf1-3d06149307eb-plugins-config-volume\") pod \"0f98b219-1d3d-47d4-aaf1-3d06149307eb\" (UID: \"0f98b219-1d3d-47d4-aaf1-3d06149307eb\") " Nov 02 23:28:11.161911 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:11.160832 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7542df71-b433-4496-b175-1797e098f9fe-kube-api-access-8nhlp\" (UniqueName: \"kubernetes.io/projected/7542df71-b433-4496-b175-1797e098f9fe-kube-api-access-8nhlp\") pod \"7542df71-b433-4496-b175-1797e098f9fe\" (UID: \"7542df71-b433-4496-b175-1797e098f9fe\") " Nov 02 23:28:11.161911 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:11.160867 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f98b219-1d3d-47d4-aaf1-3d06149307eb-kube-api-access-5bn9s\" (UniqueName: \"kubernetes.io/projected/0f98b219-1d3d-47d4-aaf1-3d06149307eb-kube-api-access-5bn9s\") pod \"0f98b219-1d3d-47d4-aaf1-3d06149307eb\" (UID: \"0f98b219-1d3d-47d4-aaf1-3d06149307eb\") " Nov 02 23:28:11.161911 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:11.161040 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7542df71-b433-4496-b175-1797e098f9fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7542df71-b433-4496-b175-1797e098f9fe-plugins-config-volume\") pod \"7542df71-b433-4496-b175-1797e098f9fe\" (UID: \"7542df71-b433-4496-b175-1797e098f9fe\") " Nov 02 23:28:11.162294 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:11.161935 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7542df71-b433-4496-b175-1797e098f9fe-plugins-config-volume" pod "7542df71-b433-4496-b175-1797e098f9fe" (UID: "7542df71-b433-4496-b175-1797e098f9fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:28:11.162478 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:11.162350 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f98b219-1d3d-47d4-aaf1-3d06149307eb-plugins-config-volume" pod "0f98b219-1d3d-47d4-aaf1-3d06149307eb" (UID: "0f98b219-1d3d-47d4-aaf1-3d06149307eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:28:11.209042 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:11.186272 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7542df71-b433-4496-b175-1797e098f9fe-kube-api-access-8nhlp" pod "7542df71-b433-4496-b175-1797e098f9fe" (UID: "7542df71-b433-4496-b175-1797e098f9fe"). InnerVolumeSpecName "kube-api-access-8nhlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:28:11.214520 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:11.213766 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f98b219-1d3d-47d4-aaf1-3d06149307eb-kube-api-access-5bn9s" pod "0f98b219-1d3d-47d4-aaf1-3d06149307eb" (UID: "0f98b219-1d3d-47d4-aaf1-3d06149307eb"). InnerVolumeSpecName "kube-api-access-5bn9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:28:11.234551 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:11.234513 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ec4a99a-59d0-422c-98f4-066c5e3eb1a1" path="/var/lib/kubelet/pods/6ec4a99a-59d0-422c-98f4-066c5e3eb1a1/volumes" Nov 02 23:28:11.262252 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:11.262212 5076 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f98b219-1d3d-47d4-aaf1-3d06149307eb-plugins-config-volume\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:11.262252 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:11.262238 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nhlp\" (UniqueName: \"kubernetes.io/projected/7542df71-b433-4496-b175-1797e098f9fe-kube-api-access-8nhlp\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:11.262252 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:11.262257 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bn9s\" (UniqueName: \"kubernetes.io/projected/0f98b219-1d3d-47d4-aaf1-3d06149307eb-kube-api-access-5bn9s\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:11.262531 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:11.262271 5076 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7542df71-b433-4496-b175-1797e098f9fe-plugins-config-volume\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:11.863398 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:11.863370 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a894c-5dbb4748-58dmg" Nov 02 23:28:11.863879 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:11.863402 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a894c-5dbb4748-58dmg" event={"ID":"7542df71-b433-4496-b175-1797e098f9fe","Type":"ContainerDied","Data":"d820e326da4599e69036ed9a3223d610ae7137c9771079a168f59bef0b6dd8c8"} Nov 02 23:28:11.863879 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:11.863467 5076 scope.go:122] "RemoveContainer" containerID="78d74cc7cfd646c8520d9a00f89449f4b8ec436e6ede934dba43ce7271a2fa14" Nov 02 23:28:11.864694 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:11.864648 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-78b54bb9f9-rtscx" event={"ID":"0f98b219-1d3d-47d4-aaf1-3d06149307eb","Type":"ContainerDied","Data":"716f10a1f14379b39e13f1edc8f6b6094531c5c27a118f3266d61d499b8fa3f7"} Nov 02 23:28:11.864694 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:11.864671 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-78b54bb9f9-rtscx" Nov 02 23:28:11.873941 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:11.873913 5076 scope.go:122] "RemoveContainer" containerID="92ce1fa84d3d751e6d06cd2640b0a5c19cfe359c5dc5755b0c6be09fa9e76f19" Nov 02 23:28:12.022777 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:12.022741 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-78b54bb9f9-rtscx"] Nov 02 23:28:12.070018 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:12.069981 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-78b54bb9f9-rtscx"] Nov 02 23:28:12.182393 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:12.182293 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6a894c-5dbb4748-58dmg"] Nov 02 23:28:12.185217 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:12.185188 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6a894c-5dbb4748-58dmg"] Nov 02 23:28:13.149420 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:13.149382 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f98b219-1d3d-47d4-aaf1-3d06149307eb" path="/var/lib/kubelet/pods/0f98b219-1d3d-47d4-aaf1-3d06149307eb/volumes" Nov 02 23:28:13.149824 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:13.149807 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7542df71-b433-4496-b175-1797e098f9fe" path="/var/lib/kubelet/pods/7542df71-b433-4496-b175-1797e098f9fe/volumes" Nov 02 23:28:14.816185 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:14.816128 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" Nov 02 23:28:14.888494 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:14.888455 5076 generic.go:356] "Generic (PLEG): container finished" podID="dfd27984-010f-441c-a3f4-553933c0529a" containerID="db7259016578f9d9120ee7dc14f0d84c043ede86b683c463e84d7f64a4a5eb63" exitCode=0 Nov 02 23:28:14.888644 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:14.888491 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" event={"ID":"dfd27984-010f-441c-a3f4-553933c0529a","Type":"ContainerDied","Data":"db7259016578f9d9120ee7dc14f0d84c043ede86b683c463e84d7f64a4a5eb63"} Nov 02 23:28:14.888644 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:14.888505 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" Nov 02 23:28:14.888644 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:14.888552 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm" event={"ID":"dfd27984-010f-441c-a3f4-553933c0529a","Type":"ContainerDied","Data":"15de44a64194381386ca89dc001c5be00985cc781a476fc7b44384c9030d06d0"} Nov 02 23:28:14.888644 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:14.888579 5076 scope.go:122] "RemoveContainer" containerID="db7259016578f9d9120ee7dc14f0d84c043ede86b683c463e84d7f64a4a5eb63" Nov 02 23:28:14.898448 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:14.898421 5076 scope.go:122] "RemoveContainer" containerID="70c2c4ebc1f046f648bf9e4350e65bc41dfb010749d084db3dc8cff0ef6e30c4" Nov 02 23:28:14.907190 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:14.907171 5076 scope.go:122] "RemoveContainer" containerID="db7259016578f9d9120ee7dc14f0d84c043ede86b683c463e84d7f64a4a5eb63" Nov 02 23:28:14.907421 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:28:14.907398 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db7259016578f9d9120ee7dc14f0d84c043ede86b683c463e84d7f64a4a5eb63\": container with ID starting with db7259016578f9d9120ee7dc14f0d84c043ede86b683c463e84d7f64a4a5eb63 not found: ID does not exist" containerID="db7259016578f9d9120ee7dc14f0d84c043ede86b683c463e84d7f64a4a5eb63" Nov 02 23:28:14.907502 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:14.907429 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db7259016578f9d9120ee7dc14f0d84c043ede86b683c463e84d7f64a4a5eb63"} err="failed to get container status \"db7259016578f9d9120ee7dc14f0d84c043ede86b683c463e84d7f64a4a5eb63\": rpc error: code = NotFound desc = could not find container \"db7259016578f9d9120ee7dc14f0d84c043ede86b683c463e84d7f64a4a5eb63\": container with ID starting with db7259016578f9d9120ee7dc14f0d84c043ede86b683c463e84d7f64a4a5eb63 not found: ID does not exist" Nov 02 23:28:14.907502 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:14.907448 5076 scope.go:122] "RemoveContainer" containerID="70c2c4ebc1f046f648bf9e4350e65bc41dfb010749d084db3dc8cff0ef6e30c4" Nov 02 23:28:14.907660 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:28:14.907641 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70c2c4ebc1f046f648bf9e4350e65bc41dfb010749d084db3dc8cff0ef6e30c4\": container with ID starting with 70c2c4ebc1f046f648bf9e4350e65bc41dfb010749d084db3dc8cff0ef6e30c4 not found: ID does not exist" containerID="70c2c4ebc1f046f648bf9e4350e65bc41dfb010749d084db3dc8cff0ef6e30c4" Nov 02 23:28:14.907706 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:14.907669 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70c2c4ebc1f046f648bf9e4350e65bc41dfb010749d084db3dc8cff0ef6e30c4"} err="failed to get container status \"70c2c4ebc1f046f648bf9e4350e65bc41dfb010749d084db3dc8cff0ef6e30c4\": rpc error: code = NotFound desc = could not find container \"70c2c4ebc1f046f648bf9e4350e65bc41dfb010749d084db3dc8cff0ef6e30c4\": container with ID starting with 70c2c4ebc1f046f648bf9e4350e65bc41dfb010749d084db3dc8cff0ef6e30c4 not found: ID does not exist" Nov 02 23:28:14.912085 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:14.912063 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd27984-010f-441c-a3f4-553933c0529a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd27984-010f-441c-a3f4-553933c0529a-credential-socket\") pod \"dfd27984-010f-441c-a3f4-553933c0529a\" (UID: \"dfd27984-010f-441c-a3f4-553933c0529a\") " Nov 02 23:28:14.912173 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:14.912109 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd27984-010f-441c-a3f4-553933c0529a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfd27984-010f-441c-a3f4-553933c0529a-workload-certs\") pod \"dfd27984-010f-441c-a3f4-553933c0529a\" (UID: \"dfd27984-010f-441c-a3f4-553933c0529a\") " Nov 02 23:28:14.912173 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:14.912162 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd27984-010f-441c-a3f4-553933c0529a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfd27984-010f-441c-a3f4-553933c0529a-istio-data\") pod \"dfd27984-010f-441c-a3f4-553933c0529a\" (UID: \"dfd27984-010f-441c-a3f4-553933c0529a\") " Nov 02 23:28:14.912328 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:14.912205 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfd27984-010f-441c-a3f4-553933c0529a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfd27984-010f-441c-a3f4-553933c0529a-istiod-ca-cert\") pod \"dfd27984-010f-441c-a3f4-553933c0529a\" (UID: \"dfd27984-010f-441c-a3f4-553933c0529a\") " Nov 02 23:28:14.912328 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:14.912268 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd27984-010f-441c-a3f4-553933c0529a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd27984-010f-441c-a3f4-553933c0529a-workload-socket\") pod \"dfd27984-010f-441c-a3f4-553933c0529a\" (UID: \"dfd27984-010f-441c-a3f4-553933c0529a\") " Nov 02 23:28:14.912328 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:14.912297 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfd27984-010f-441c-a3f4-553933c0529a-istio-token\" (UniqueName: \"kubernetes.io/projected/dfd27984-010f-441c-a3f4-553933c0529a-istio-token\") pod \"dfd27984-010f-441c-a3f4-553933c0529a\" (UID: \"dfd27984-010f-441c-a3f4-553933c0529a\") " Nov 02 23:28:14.912328 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:14.912318 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd27984-010f-441c-a3f4-553933c0529a-credential-socket" pod "dfd27984-010f-441c-a3f4-553933c0529a" (UID: "dfd27984-010f-441c-a3f4-553933c0529a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:14.912572 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:14.912408 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd27984-010f-441c-a3f4-553933c0529a-workload-certs" pod "dfd27984-010f-441c-a3f4-553933c0529a" (UID: "dfd27984-010f-441c-a3f4-553933c0529a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:14.912572 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:14.912429 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd27984-010f-441c-a3f4-553933c0529a-istio-data" pod "dfd27984-010f-441c-a3f4-553933c0529a" (UID: "dfd27984-010f-441c-a3f4-553933c0529a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:14.912572 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:14.912481 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd27984-010f-441c-a3f4-553933c0529a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfd27984-010f-441c-a3f4-553933c0529a-istio-envoy\") pod \"dfd27984-010f-441c-a3f4-553933c0529a\" (UID: \"dfd27984-010f-441c-a3f4-553933c0529a\") " Nov 02 23:28:14.912572 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:14.912512 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd27984-010f-441c-a3f4-553933c0529a-workload-socket" pod "dfd27984-010f-441c-a3f4-553933c0529a" (UID: "dfd27984-010f-441c-a3f4-553933c0529a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:14.912572 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:14.912515 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfd27984-010f-441c-a3f4-553933c0529a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfd27984-010f-441c-a3f4-553933c0529a-istio-podinfo\") pod \"dfd27984-010f-441c-a3f4-553933c0529a\" (UID: \"dfd27984-010f-441c-a3f4-553933c0529a\") " Nov 02 23:28:14.912867 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:14.912574 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfd27984-010f-441c-a3f4-553933c0529a-kube-api-access-f2jbn\" (UniqueName: \"kubernetes.io/projected/dfd27984-010f-441c-a3f4-553933c0529a-kube-api-access-f2jbn\") pod \"dfd27984-010f-441c-a3f4-553933c0529a\" (UID: \"dfd27984-010f-441c-a3f4-553933c0529a\") " Nov 02 23:28:14.912867 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:14.912630 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfd27984-010f-441c-a3f4-553933c0529a-istiod-ca-cert" pod "dfd27984-010f-441c-a3f4-553933c0529a" (UID: "dfd27984-010f-441c-a3f4-553933c0529a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:28:14.913205 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:14.913183 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd27984-010f-441c-a3f4-553933c0529a-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:14.913358 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:14.913333 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfd27984-010f-441c-a3f4-553933c0529a-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:14.913479 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:14.913359 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfd27984-010f-441c-a3f4-553933c0529a-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:14.913479 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:14.913376 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfd27984-010f-441c-a3f4-553933c0529a-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:14.913479 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:14.913391 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd27984-010f-441c-a3f4-553933c0529a-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:14.914683 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:14.914655 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd27984-010f-441c-a3f4-553933c0529a-istio-envoy" pod "dfd27984-010f-441c-a3f4-553933c0529a" (UID: "dfd27984-010f-441c-a3f4-553933c0529a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:14.914683 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:14.914666 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfd27984-010f-441c-a3f4-553933c0529a-istio-podinfo" pod "dfd27984-010f-441c-a3f4-553933c0529a" (UID: "dfd27984-010f-441c-a3f4-553933c0529a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:28:14.914825 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:14.914670 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfd27984-010f-441c-a3f4-553933c0529a-istio-token" pod "dfd27984-010f-441c-a3f4-553933c0529a" (UID: "dfd27984-010f-441c-a3f4-553933c0529a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:28:14.914825 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:14.914774 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfd27984-010f-441c-a3f4-553933c0529a-kube-api-access-f2jbn" pod "dfd27984-010f-441c-a3f4-553933c0529a" (UID: "dfd27984-010f-441c-a3f4-553933c0529a"). InnerVolumeSpecName "kube-api-access-f2jbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:28:15.014492 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:15.014452 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfd27984-010f-441c-a3f4-553933c0529a-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:15.014492 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:15.014484 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfd27984-010f-441c-a3f4-553933c0529a-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:15.014492 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:15.014497 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfd27984-010f-441c-a3f4-553933c0529a-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:15.014779 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:15.014511 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2jbn\" (UniqueName: \"kubernetes.io/projected/dfd27984-010f-441c-a3f4-553933c0529a-kube-api-access-f2jbn\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:15.302402 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:15.302366 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm"] Nov 02 23:28:15.307412 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:15.307384 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b57963-openshift-default-5bfd979f57-jqfqm"] Nov 02 23:28:16.879329 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:16.879295 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97"] Nov 02 23:28:16.879801 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:16.879516 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97" podUID="21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8" containerName="maas-api" containerID="cri-o://855069e5aeea29b8df92a0d244ec3b1dbde9001621bff8637e2f5f958d88e312" gracePeriod=30 Nov 02 23:28:17.151308 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:17.151265 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfd27984-010f-441c-a3f4-553933c0529a" path="/var/lib/kubelet/pods/dfd27984-010f-441c-a3f4-553933c0529a/volumes" Nov 02 23:28:17.190959 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:17.190511 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97" Nov 02 23:28:17.247687 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:17.247646 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8-maas-api-tls\") pod \"21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8\" (UID: \"21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8\") " Nov 02 23:28:17.247925 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:17.247704 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8-metrics-certs\") pod \"21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8\" (UID: \"21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8\") " Nov 02 23:28:17.247925 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:17.247745 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8-openshift-service-ca\") pod \"21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8\" (UID: \"21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8\") " Nov 02 23:28:17.248046 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:17.247928 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8-kube-api-access-qx6nb\" (UniqueName: \"kubernetes.io/projected/21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8-kube-api-access-qx6nb\") pod \"21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8\" (UID: \"21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8\") " Nov 02 23:28:17.249300 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:17.249269 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8-openshift-service-ca" pod "21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8" (UID: "21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:28:17.261051 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:17.261009 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8-maas-api-tls" pod "21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8" (UID: "21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:28:17.261209 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:17.261129 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8-kube-api-access-qx6nb" pod "21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8" (UID: "21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8"). InnerVolumeSpecName "kube-api-access-qx6nb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:28:17.261209 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:17.261143 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8-metrics-certs" pod "21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8" (UID: "21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:28:17.369002 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:17.368438 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:17.369002 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:17.368479 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:17.369002 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:17.368494 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx6nb\" (UniqueName: \"kubernetes.io/projected/21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8-kube-api-access-qx6nb\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:17.369002 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:17.368510 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:17.958910 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:17.950583 5076 generic.go:356] "Generic (PLEG): container finished" podID="21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8" containerID="855069e5aeea29b8df92a0d244ec3b1dbde9001621bff8637e2f5f958d88e312" exitCode=0 Nov 02 23:28:17.958910 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:17.950776 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97" event={"ID":"21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8","Type":"ContainerDied","Data":"855069e5aeea29b8df92a0d244ec3b1dbde9001621bff8637e2f5f958d88e312"} Nov 02 23:28:17.958910 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:17.950825 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97" event={"ID":"21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8","Type":"ContainerDied","Data":"88175317708cae229e792c3a7c8544c5767f0e5478ef318d3c612320eddc2d3a"} Nov 02 23:28:17.958910 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:17.950849 5076 scope.go:122] "RemoveContainer" containerID="855069e5aeea29b8df92a0d244ec3b1dbde9001621bff8637e2f5f958d88e312" Nov 02 23:28:17.958910 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:17.956045 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97" Nov 02 23:28:17.997475 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:17.997445 5076 scope.go:122] "RemoveContainer" containerID="855069e5aeea29b8df92a0d244ec3b1dbde9001621bff8637e2f5f958d88e312" Nov 02 23:28:17.997863 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:28:17.997841 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"855069e5aeea29b8df92a0d244ec3b1dbde9001621bff8637e2f5f958d88e312\": container with ID starting with 855069e5aeea29b8df92a0d244ec3b1dbde9001621bff8637e2f5f958d88e312 not found: ID does not exist" containerID="855069e5aeea29b8df92a0d244ec3b1dbde9001621bff8637e2f5f958d88e312" Nov 02 23:28:17.997950 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:17.997876 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"855069e5aeea29b8df92a0d244ec3b1dbde9001621bff8637e2f5f958d88e312"} err="failed to get container status \"855069e5aeea29b8df92a0d244ec3b1dbde9001621bff8637e2f5f958d88e312\": rpc error: code = NotFound desc = could not find container \"855069e5aeea29b8df92a0d244ec3b1dbde9001621bff8637e2f5f958d88e312\": container with ID starting with 855069e5aeea29b8df92a0d244ec3b1dbde9001621bff8637e2f5f958d88e312 not found: ID does not exist" Nov 02 23:28:18.135617 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:18.135572 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97"] Nov 02 23:28:18.183000 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:18.182962 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-443fdf-55c64ff6bb-fbj97"] Nov 02 23:28:19.148599 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:19.148556 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8" path="/var/lib/kubelet/pods/21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8/volumes" Nov 02 23:28:22.042878 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:22.042841 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn"] Nov 02 23:28:22.043343 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:22.043142 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" podUID="a1632e0b-a576-4db6-ba82-dc11e89a7c00" containerName="istio-proxy" containerID="cri-o://cdfd791b71e12d995d72723583b6310eb431817997838a103b2b84adac141348" gracePeriod=30 Nov 02 23:28:27.317614 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:27.317556 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" Nov 02 23:28:27.397950 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:27.397910 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1632e0b-a576-4db6-ba82-dc11e89a7c00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1632e0b-a576-4db6-ba82-dc11e89a7c00-istio-envoy\") pod \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\" (UID: \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\") " Nov 02 23:28:27.398121 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:27.397962 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1632e0b-a576-4db6-ba82-dc11e89a7c00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1632e0b-a576-4db6-ba82-dc11e89a7c00-credential-socket\") pod \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\" (UID: \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\") " Nov 02 23:28:27.398121 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:27.397993 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1632e0b-a576-4db6-ba82-dc11e89a7c00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1632e0b-a576-4db6-ba82-dc11e89a7c00-istio-data\") pod \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\" (UID: \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\") " Nov 02 23:28:27.398121 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:27.398025 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1632e0b-a576-4db6-ba82-dc11e89a7c00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1632e0b-a576-4db6-ba82-dc11e89a7c00-workload-socket\") pod \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\" (UID: \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\") " Nov 02 23:28:27.398121 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:27.398083 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1632e0b-a576-4db6-ba82-dc11e89a7c00-kube-api-access-hvmhq\" (UniqueName: \"kubernetes.io/projected/a1632e0b-a576-4db6-ba82-dc11e89a7c00-kube-api-access-hvmhq\") pod \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\" (UID: \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\") " Nov 02 23:28:27.398121 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:27.398113 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1632e0b-a576-4db6-ba82-dc11e89a7c00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1632e0b-a576-4db6-ba82-dc11e89a7c00-istio-podinfo\") pod \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\" (UID: \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\") " Nov 02 23:28:27.398389 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:27.398143 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1632e0b-a576-4db6-ba82-dc11e89a7c00-istio-token\" (UniqueName: \"kubernetes.io/projected/a1632e0b-a576-4db6-ba82-dc11e89a7c00-istio-token\") pod \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\" (UID: \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\") " Nov 02 23:28:27.398389 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:27.398173 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1632e0b-a576-4db6-ba82-dc11e89a7c00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1632e0b-a576-4db6-ba82-dc11e89a7c00-istiod-ca-cert\") pod \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\" (UID: \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\") " Nov 02 23:28:27.398389 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:27.398206 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1632e0b-a576-4db6-ba82-dc11e89a7c00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1632e0b-a576-4db6-ba82-dc11e89a7c00-workload-certs\") pod \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\" (UID: \"a1632e0b-a576-4db6-ba82-dc11e89a7c00\") " Nov 02 23:28:27.398389 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:27.398308 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1632e0b-a576-4db6-ba82-dc11e89a7c00-credential-socket" pod "a1632e0b-a576-4db6-ba82-dc11e89a7c00" (UID: "a1632e0b-a576-4db6-ba82-dc11e89a7c00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:27.398389 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:27.398317 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1632e0b-a576-4db6-ba82-dc11e89a7c00-istio-data" pod "a1632e0b-a576-4db6-ba82-dc11e89a7c00" (UID: "a1632e0b-a576-4db6-ba82-dc11e89a7c00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:27.398657 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:27.398547 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1632e0b-a576-4db6-ba82-dc11e89a7c00-workload-socket" pod "a1632e0b-a576-4db6-ba82-dc11e89a7c00" (UID: "a1632e0b-a576-4db6-ba82-dc11e89a7c00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:27.398719 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:27.398644 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1632e0b-a576-4db6-ba82-dc11e89a7c00-istiod-ca-cert" pod "a1632e0b-a576-4db6-ba82-dc11e89a7c00" (UID: "a1632e0b-a576-4db6-ba82-dc11e89a7c00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:28:27.398781 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:27.398715 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1632e0b-a576-4db6-ba82-dc11e89a7c00-workload-certs" pod "a1632e0b-a576-4db6-ba82-dc11e89a7c00" (UID: "a1632e0b-a576-4db6-ba82-dc11e89a7c00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:27.398987 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:27.398960 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1632e0b-a576-4db6-ba82-dc11e89a7c00-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:27.398987 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:27.398987 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1632e0b-a576-4db6-ba82-dc11e89a7c00-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:27.399194 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:27.399001 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1632e0b-a576-4db6-ba82-dc11e89a7c00-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:27.399194 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:27.399015 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1632e0b-a576-4db6-ba82-dc11e89a7c00-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:27.399194 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:27.399029 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1632e0b-a576-4db6-ba82-dc11e89a7c00-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:27.400365 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:27.400335 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1632e0b-a576-4db6-ba82-dc11e89a7c00-istio-token" pod "a1632e0b-a576-4db6-ba82-dc11e89a7c00" (UID: "a1632e0b-a576-4db6-ba82-dc11e89a7c00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:28:27.400467 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:27.400388 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1632e0b-a576-4db6-ba82-dc11e89a7c00-kube-api-access-hvmhq" pod "a1632e0b-a576-4db6-ba82-dc11e89a7c00" (UID: "a1632e0b-a576-4db6-ba82-dc11e89a7c00"). InnerVolumeSpecName "kube-api-access-hvmhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:28:27.400467 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:27.400409 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1632e0b-a576-4db6-ba82-dc11e89a7c00-istio-envoy" pod "a1632e0b-a576-4db6-ba82-dc11e89a7c00" (UID: "a1632e0b-a576-4db6-ba82-dc11e89a7c00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:27.400467 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:27.400459 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1632e0b-a576-4db6-ba82-dc11e89a7c00-istio-podinfo" pod "a1632e0b-a576-4db6-ba82-dc11e89a7c00" (UID: "a1632e0b-a576-4db6-ba82-dc11e89a7c00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:28:27.500584 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:27.500551 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvmhq\" (UniqueName: \"kubernetes.io/projected/a1632e0b-a576-4db6-ba82-dc11e89a7c00-kube-api-access-hvmhq\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:27.500584 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:27.500581 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1632e0b-a576-4db6-ba82-dc11e89a7c00-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:27.500584 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:27.500592 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1632e0b-a576-4db6-ba82-dc11e89a7c00-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:27.500584 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:27.500600 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1632e0b-a576-4db6-ba82-dc11e89a7c00-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:28.033799 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:28.033766 5076 generic.go:356] "Generic (PLEG): container finished" podID="a1632e0b-a576-4db6-ba82-dc11e89a7c00" containerID="cdfd791b71e12d995d72723583b6310eb431817997838a103b2b84adac141348" exitCode=0 Nov 02 23:28:28.033970 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:28.033815 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" Nov 02 23:28:28.033970 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:28.033854 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" event={"ID":"a1632e0b-a576-4db6-ba82-dc11e89a7c00","Type":"ContainerDied","Data":"cdfd791b71e12d995d72723583b6310eb431817997838a103b2b84adac141348"} Nov 02 23:28:28.033970 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:28.033928 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn" event={"ID":"a1632e0b-a576-4db6-ba82-dc11e89a7c00","Type":"ContainerDied","Data":"198050dfbd6fd0d119dccff87bb6f702582c76253edfd7335b9beada5064b6e7"} Nov 02 23:28:28.033970 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:28.033954 5076 scope.go:122] "RemoveContainer" containerID="cdfd791b71e12d995d72723583b6310eb431817997838a103b2b84adac141348" Nov 02 23:28:28.045058 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:28.045040 5076 scope.go:122] "RemoveContainer" containerID="12e662bb18e88e5acd93940191e506762a5c8d60d33f2c366c5d70e535342593" Nov 02 23:28:28.053794 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:28.053776 5076 scope.go:122] "RemoveContainer" containerID="cdfd791b71e12d995d72723583b6310eb431817997838a103b2b84adac141348" Nov 02 23:28:28.054077 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:28:28.054056 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdfd791b71e12d995d72723583b6310eb431817997838a103b2b84adac141348\": container with ID starting with cdfd791b71e12d995d72723583b6310eb431817997838a103b2b84adac141348 not found: ID does not exist" containerID="cdfd791b71e12d995d72723583b6310eb431817997838a103b2b84adac141348" Nov 02 23:28:28.054155 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:28.054090 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdfd791b71e12d995d72723583b6310eb431817997838a103b2b84adac141348"} err="failed to get container status \"cdfd791b71e12d995d72723583b6310eb431817997838a103b2b84adac141348\": rpc error: code = NotFound desc = could not find container \"cdfd791b71e12d995d72723583b6310eb431817997838a103b2b84adac141348\": container with ID starting with cdfd791b71e12d995d72723583b6310eb431817997838a103b2b84adac141348 not found: ID does not exist" Nov 02 23:28:28.054155 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:28.054117 5076 scope.go:122] "RemoveContainer" containerID="12e662bb18e88e5acd93940191e506762a5c8d60d33f2c366c5d70e535342593" Nov 02 23:28:28.054365 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:28:28.054346 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12e662bb18e88e5acd93940191e506762a5c8d60d33f2c366c5d70e535342593\": container with ID starting with 12e662bb18e88e5acd93940191e506762a5c8d60d33f2c366c5d70e535342593 not found: ID does not exist" containerID="12e662bb18e88e5acd93940191e506762a5c8d60d33f2c366c5d70e535342593" Nov 02 23:28:28.054420 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:28.054373 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12e662bb18e88e5acd93940191e506762a5c8d60d33f2c366c5d70e535342593"} err="failed to get container status \"12e662bb18e88e5acd93940191e506762a5c8d60d33f2c366c5d70e535342593\": rpc error: code = NotFound desc = could not find container \"12e662bb18e88e5acd93940191e506762a5c8d60d33f2c366c5d70e535342593\": container with ID starting with 12e662bb18e88e5acd93940191e506762a5c8d60d33f2c366c5d70e535342593 not found: ID does not exist" Nov 02 23:28:28.191031 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:28.190996 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn"] Nov 02 23:28:28.195899 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:28.195860 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-6a894c-openshift-default-dc7dfc8b9-hs5jn"] Nov 02 23:28:29.148445 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:29.148411 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1632e0b-a576-4db6-ba82-dc11e89a7c00" path="/var/lib/kubelet/pods/a1632e0b-a576-4db6-ba82-dc11e89a7c00/volumes" Nov 02 23:28:30.220703 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.220666 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-7cd9d669b7-92tmp"] Nov 02 23:28:30.221225 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.221210 5076 cpu_manager.go:414] "containerMap: removing container" podUID="a1632e0b-a576-4db6-ba82-dc11e89a7c00" containerName="istio-proxy" Nov 02 23:28:30.221277 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.221226 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1632e0b-a576-4db6-ba82-dc11e89a7c00" containerName="istio-proxy" Nov 02 23:28:30.221277 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.221234 5076 cpu_manager.go:414] "containerMap: removing container" podUID="7542df71-b433-4496-b175-1797e098f9fe" containerName="payload-processing" Nov 02 23:28:30.221277 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.221240 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7542df71-b433-4496-b175-1797e098f9fe" containerName="payload-processing" Nov 02 23:28:30.221277 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.221254 5076 cpu_manager.go:414] "containerMap: removing container" podUID="dfd27984-010f-441c-a3f4-553933c0529a" containerName="istio-proxy" Nov 02 23:28:30.221277 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.221259 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfd27984-010f-441c-a3f4-553933c0529a" containerName="istio-proxy" Nov 02 23:28:30.221277 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.221265 5076 cpu_manager.go:414] "containerMap: removing container" podUID="21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8" containerName="maas-api" Nov 02 23:28:30.221277 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.221269 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8" containerName="maas-api" Nov 02 23:28:30.221277 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.221279 5076 cpu_manager.go:414] "containerMap: removing container" podUID="0f98b219-1d3d-47d4-aaf1-3d06149307eb" containerName="payload-pre-processing" Nov 02 23:28:30.221521 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.221283 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f98b219-1d3d-47d4-aaf1-3d06149307eb" containerName="payload-pre-processing" Nov 02 23:28:30.221521 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.221295 5076 cpu_manager.go:414] "containerMap: removing container" podUID="dfd27984-010f-441c-a3f4-553933c0529a" containerName="istio-proxy" Nov 02 23:28:30.221521 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.221300 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfd27984-010f-441c-a3f4-553933c0529a" containerName="istio-proxy" Nov 02 23:28:30.221521 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.221310 5076 cpu_manager.go:414] "containerMap: removing container" podUID="a1632e0b-a576-4db6-ba82-dc11e89a7c00" containerName="istio-proxy" Nov 02 23:28:30.221521 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.221315 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1632e0b-a576-4db6-ba82-dc11e89a7c00" containerName="istio-proxy" Nov 02 23:28:30.221521 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.221327 5076 cpu_manager.go:414] "containerMap: removing container" podUID="6ec4a99a-59d0-422c-98f4-066c5e3eb1a1" containerName="maas-api" Nov 02 23:28:30.221521 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.221331 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ec4a99a-59d0-422c-98f4-066c5e3eb1a1" containerName="maas-api" Nov 02 23:28:30.221521 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.221465 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f98b219-1d3d-47d4-aaf1-3d06149307eb" containerName="payload-pre-processing" Nov 02 23:28:30.221521 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.221477 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfd27984-010f-441c-a3f4-553933c0529a" containerName="istio-proxy" Nov 02 23:28:30.221521 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.221496 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1632e0b-a576-4db6-ba82-dc11e89a7c00" containerName="istio-proxy" Nov 02 23:28:30.221521 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.221505 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ec4a99a-59d0-422c-98f4-066c5e3eb1a1" containerName="maas-api" Nov 02 23:28:30.221521 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.221516 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="7542df71-b433-4496-b175-1797e098f9fe" containerName="payload-processing" Nov 02 23:28:30.221521 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.221523 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="21ba1f08-6cec-4c24-8fa0-745fdf3f3ce8" containerName="maas-api" Nov 02 23:28:30.224570 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.224548 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-7cd9d669b7-92tmp" Nov 02 23:28:30.276982 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.276953 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-7cd9d669b7-92tmp"] Nov 02 23:28:30.320809 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.320780 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-7fdfca\"" Nov 02 23:28:30.320809 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.320785 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-7fdfca-dockercfg-x662m\"" Nov 02 23:28:30.333357 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.333327 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wf9p\" (UniqueName: \"kubernetes.io/projected/cbb5f075-306d-4826-a938-5363979473cc-kube-api-access-7wf9p\") pod \"payload-pre-processing-e2e-ipp-b-7fdfca-7cd9d669b7-92tmp\" (UID: \"cbb5f075-306d-4826-a938-5363979473cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-7cd9d669b7-92tmp" Nov 02 23:28:30.333466 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.333375 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbb5f075-306d-4826-a938-5363979473cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7fdfca-7cd9d669b7-92tmp\" (UID: \"cbb5f075-306d-4826-a938-5363979473cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-7cd9d669b7-92tmp" Nov 02 23:28:30.435003 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.434968 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wf9p\" (UniqueName: \"kubernetes.io/projected/cbb5f075-306d-4826-a938-5363979473cc-kube-api-access-7wf9p\") pod \"payload-pre-processing-e2e-ipp-b-7fdfca-7cd9d669b7-92tmp\" (UID: \"cbb5f075-306d-4826-a938-5363979473cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-7cd9d669b7-92tmp" Nov 02 23:28:30.435164 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.435046 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbb5f075-306d-4826-a938-5363979473cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7fdfca-7cd9d669b7-92tmp\" (UID: \"cbb5f075-306d-4826-a938-5363979473cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-7cd9d669b7-92tmp" Nov 02 23:28:30.435769 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.435748 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbb5f075-306d-4826-a938-5363979473cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7fdfca-7cd9d669b7-92tmp\" (UID: \"cbb5f075-306d-4826-a938-5363979473cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-7cd9d669b7-92tmp" Nov 02 23:28:30.489629 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.489548 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wf9p\" (UniqueName: \"kubernetes.io/projected/cbb5f075-306d-4826-a938-5363979473cc-kube-api-access-7wf9p\") pod \"payload-pre-processing-e2e-ipp-b-7fdfca-7cd9d669b7-92tmp\" (UID: \"cbb5f075-306d-4826-a938-5363979473cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-7cd9d669b7-92tmp" Nov 02 23:28:30.537754 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.537723 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-7cd9d669b7-92tmp" Nov 02 23:28:30.902877 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.902846 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-7cd9d669b7-92tmp"] Nov 02 23:28:30.903936 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:28:30.903907 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb5f075_306d_4826_a938_5363979473cc.slice/crio-187f6bc7aa521ad7a1e6ecda10e3af4e4b078d00cdec04ced50941feb4e74c48 WatchSource:0}: Error finding container 187f6bc7aa521ad7a1e6ecda10e3af4e4b078d00cdec04ced50941feb4e74c48: Status 404 returned error can't find the container with id 187f6bc7aa521ad7a1e6ecda10e3af4e4b078d00cdec04ced50941feb4e74c48 Nov 02 23:28:30.908842 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.908816 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-5fc65f84db-2jcf4"] Nov 02 23:28:30.909803 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.909784 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1632e0b-a576-4db6-ba82-dc11e89a7c00" containerName="istio-proxy" Nov 02 23:28:30.909837 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.909821 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfd27984-010f-441c-a3f4-553933c0529a" containerName="istio-proxy" Nov 02 23:28:30.913127 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.913107 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-5fc65f84db-2jcf4" Nov 02 23:28:30.965056 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:30.965005 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-5fc65f84db-2jcf4"] Nov 02 23:28:31.041701 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:31.041664 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/554c20cb-570b-49f7-af7a-b9eb11177564-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7fdfca-5fc65f84db-2jcf4\" (UID: \"554c20cb-570b-49f7-af7a-b9eb11177564\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-5fc65f84db-2jcf4" Nov 02 23:28:31.041920 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:31.041864 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc2tb\" (UniqueName: \"kubernetes.io/projected/554c20cb-570b-49f7-af7a-b9eb11177564-kube-api-access-lc2tb\") pod \"payload-processing-e2e-ipp-b-7fdfca-5fc65f84db-2jcf4\" (UID: \"554c20cb-570b-49f7-af7a-b9eb11177564\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-5fc65f84db-2jcf4" Nov 02 23:28:31.054782 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:31.054748 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-7cd9d669b7-92tmp" event={"ID":"cbb5f075-306d-4826-a938-5363979473cc","Type":"ContainerStarted","Data":"3330427ef8c4ab954b3693c211e9c988499ba4d05683287afcf2315cdf6fde66"} Nov 02 23:28:31.054953 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:31.054809 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-7cd9d669b7-92tmp" event={"ID":"cbb5f075-306d-4826-a938-5363979473cc","Type":"ContainerStarted","Data":"187f6bc7aa521ad7a1e6ecda10e3af4e4b078d00cdec04ced50941feb4e74c48"} Nov 02 23:28:31.054953 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:31.054838 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-7cd9d669b7-92tmp" Nov 02 23:28:31.143044 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:31.143008 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc2tb\" (UniqueName: \"kubernetes.io/projected/554c20cb-570b-49f7-af7a-b9eb11177564-kube-api-access-lc2tb\") pod \"payload-processing-e2e-ipp-b-7fdfca-5fc65f84db-2jcf4\" (UID: \"554c20cb-570b-49f7-af7a-b9eb11177564\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-5fc65f84db-2jcf4" Nov 02 23:28:31.143218 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:31.143130 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/554c20cb-570b-49f7-af7a-b9eb11177564-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7fdfca-5fc65f84db-2jcf4\" (UID: \"554c20cb-570b-49f7-af7a-b9eb11177564\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-5fc65f84db-2jcf4" Nov 02 23:28:31.143829 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:31.143804 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/554c20cb-570b-49f7-af7a-b9eb11177564-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7fdfca-5fc65f84db-2jcf4\" (UID: \"554c20cb-570b-49f7-af7a-b9eb11177564\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-5fc65f84db-2jcf4" Nov 02 23:28:31.164628 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:31.164535 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-7cd9d669b7-92tmp" podStartSLOduration=1.164522756 podStartE2EDuration="1.164522756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:28:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:28:31.162774004 +0000 UTC m=+2160.131305778" watchObservedRunningTime="2026-11-02 23:28:31.164522756 +0000 UTC m=+2160.133054568" Nov 02 23:28:31.197015 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:31.196986 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc2tb\" (UniqueName: \"kubernetes.io/projected/554c20cb-570b-49f7-af7a-b9eb11177564-kube-api-access-lc2tb\") pod \"payload-processing-e2e-ipp-b-7fdfca-5fc65f84db-2jcf4\" (UID: \"554c20cb-570b-49f7-af7a-b9eb11177564\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-5fc65f84db-2jcf4" Nov 02 23:28:31.253794 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:31.253764 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-5fc65f84db-2jcf4" Nov 02 23:28:31.640844 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:31.640811 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-5fc65f84db-2jcf4"] Nov 02 23:28:31.642038 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:28:31.642005 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod554c20cb_570b_49f7_af7a_b9eb11177564.slice/crio-34e8b2f2c13f713be11d90520b81be3f30dd51a31f9d10e0f0f62868839a5fc5 WatchSource:0}: Error finding container 34e8b2f2c13f713be11d90520b81be3f30dd51a31f9d10e0f0f62868839a5fc5: Status 404 returned error can't find the container with id 34e8b2f2c13f713be11d90520b81be3f30dd51a31f9d10e0f0f62868839a5fc5 Nov 02 23:28:32.064714 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:32.064675 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-5fc65f84db-2jcf4" event={"ID":"554c20cb-570b-49f7-af7a-b9eb11177564","Type":"ContainerStarted","Data":"4fc82e223fe829fe5bca6b552c7882239df85fbf3cf56673ad0b0bdf7557fbac"} Nov 02 23:28:32.064937 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:32.064741 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-5fc65f84db-2jcf4" event={"ID":"554c20cb-570b-49f7-af7a-b9eb11177564","Type":"ContainerStarted","Data":"34e8b2f2c13f713be11d90520b81be3f30dd51a31f9d10e0f0f62868839a5fc5"} Nov 02 23:28:32.065077 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:32.065058 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-5fc65f84db-2jcf4" Nov 02 23:28:32.176657 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:32.176539 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-5fc65f84db-2jcf4" podStartSLOduration=2.176524371 podStartE2EDuration="2.176524371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:28:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:28:32.173122681 +0000 UTC m=+2161.141654468" watchObservedRunningTime="2026-11-02 23:28:32.176524371 +0000 UTC m=+2161.145056155" Nov 02 23:28:39.888941 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:28:39.888873 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:28:42.068060 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:42.068029 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-7cd9d669b7-92tmp" Nov 02 23:28:43.074351 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:43.074318 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-5fc65f84db-2jcf4" Nov 02 23:28:51.504969 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:51.504245 5076 patch_prober.go:29] interesting pod/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": read tcp 10.132.0.2:38504->10.132.0.133:15021: read: connection reset by peer" start-of-body= Nov 02 23:28:51.504969 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:51.504320 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" podUID="6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": read tcp 10.132.0.2:38504->10.132.0.133:15021: read: connection reset by peer" Nov 02 23:28:52.220501 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:52.220465 5076 generic.go:356] "Generic (PLEG): container finished" podID="6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91" containerID="e5d4bf19db2a1293fb208a5ebda6a1e629e4289d83626d9b996a3e7fe2fbb601" exitCode=0 Nov 02 23:28:52.220682 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:52.220525 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" event={"ID":"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91","Type":"ContainerDied","Data":"e5d4bf19db2a1293fb208a5ebda6a1e629e4289d83626d9b996a3e7fe2fbb601"} Nov 02 23:28:52.220934 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:52.220919 5076 scope.go:122] "RemoveContainer" containerID="e5d4bf19db2a1293fb208a5ebda6a1e629e4289d83626d9b996a3e7fe2fbb601" Nov 02 23:28:52.222495 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:52.222468 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:28:52.222568 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:52.222534 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:28:52.222568 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:52.222564 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:28:53.240779 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:53.240747 5076 generic.go:356] "Generic (PLEG): container finished" podID="ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9" containerID="25d296194a17393f8f3e7d15b994b96d25cbb18c18540c4c94e0ecca5d3da43a" exitCode=0 Nov 02 23:28:53.241306 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:53.240824 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" event={"ID":"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9","Type":"ContainerDied","Data":"25d296194a17393f8f3e7d15b994b96d25cbb18c18540c4c94e0ecca5d3da43a"} Nov 02 23:28:53.241306 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:53.241293 5076 scope.go:122] "RemoveContainer" containerID="25d296194a17393f8f3e7d15b994b96d25cbb18c18540c4c94e0ecca5d3da43a" Nov 02 23:28:53.242919 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:53.242878 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:28:53.242919 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:53.242911 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" event={"ID":"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91","Type":"ContainerStarted","Data":"8a88dbbc1feb4213a82a24bd49011011a93468b93bc25bf7580d117753d65525"} Nov 02 23:28:53.243065 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:53.242979 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:28:53.243065 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:53.243011 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:28:53.540016 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:53.532070 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" Nov 02 23:28:53.540016 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:53.533397 5076 patch_prober.go:29] interesting pod/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 23:28:53.540016 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:53.533453 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" podUID="6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 23:28:54.269749 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:54.269710 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" event={"ID":"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9","Type":"ContainerStarted","Data":"b76d3473da76180aac00b85fba07f1352d9e22291b807744217a387f1d564641"} Nov 02 23:28:54.535992 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:54.531456 5076 patch_prober.go:29] interesting pod/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 23:28:54.535992 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:54.531540 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" podUID="6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 23:28:55.007916 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:54.997264 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" Nov 02 23:28:55.046915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:55.043384 5076 patch_prober.go:29] interesting pod/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 02 23:28:55.046915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:55.043458 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" podUID="ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 02 23:28:55.539915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:55.533357 5076 patch_prober.go:29] interesting pod/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 23:28:55.539915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:55.533421 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" podUID="6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 23:28:55.977484 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:55.977431 5076 patch_prober.go:29] interesting pod/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 02 23:28:55.977673 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:55.977505 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" podUID="ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 02 23:28:56.287002 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:56.286927 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" Nov 02 23:28:56.531916 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:56.531391 5076 patch_prober.go:29] interesting pod/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 23:28:56.531916 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:56.531474 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" podUID="6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 23:28:56.976742 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:56.976694 5076 patch_prober.go:29] interesting pod/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 02 23:28:56.977289 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:56.976771 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" podUID="ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 02 23:28:57.531323 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:57.531282 5076 patch_prober.go:29] interesting pod/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 23:28:57.531630 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:57.531598 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" podUID="6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 23:28:57.976673 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:57.976643 5076 patch_prober.go:29] interesting pod/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 02 23:28:57.976932 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:57.976882 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" podUID="ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 02 23:28:58.602185 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:58.602156 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" Nov 02 23:28:58.602990 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:58.602959 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" Nov 02 23:28:58.603682 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:58.603662 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" Nov 02 23:28:58.977107 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:58.977069 5076 patch_prober.go:29] interesting pod/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 02 23:28:58.977595 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:58.977129 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" podUID="ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 02 23:28:59.981573 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:59.981531 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" Nov 02 23:28:59.982667 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:28:59.982643 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" Nov 02 23:29:09.884686 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:29:09.884643 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:29:30.407934 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:30.407876 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4"] Nov 02 23:29:30.408332 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:30.408167 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" podUID="8c5807a1-409d-4188-abb7-1bf3bfb8e07d" containerName="istio-proxy" containerID="cri-o://c7800ec2dfb7b4ff4805a5b9a9a863134b456ee5f343daaea2c49ab10487e1e3" gracePeriod=30 Nov 02 23:29:30.868744 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:30.868707 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps"] Nov 02 23:29:30.868973 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:30.868914 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps" podUID="b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3" containerName="maas-api" containerID="cri-o://c28e40e0f1feefa98b64ec8581c52360f57c4732603a6cf596306a9e1cccba6f" gracePeriod=30 Nov 02 23:29:31.167083 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.167056 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps" Nov 02 23:29:31.262217 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.262183 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3-maas-api-tls\") pod \"b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3\" (UID: \"b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3\") " Nov 02 23:29:31.262390 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.262336 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3-openshift-service-ca\") pod \"b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3\" (UID: \"b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3\") " Nov 02 23:29:31.262458 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.262407 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3-metrics-certs\") pod \"b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3\" (UID: \"b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3\") " Nov 02 23:29:31.262458 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.262449 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3-kube-api-access-9gsq6\" (UniqueName: \"kubernetes.io/projected/b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3-kube-api-access-9gsq6\") pod \"b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3\" (UID: \"b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3\") " Nov 02 23:29:31.262736 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.262707 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3-openshift-service-ca" pod "b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3" (UID: "b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:29:31.263207 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.263182 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:29:31.264556 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.264522 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3-maas-api-tls" pod "b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3" (UID: "b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:29:31.264669 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.264602 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3-metrics-certs" pod "b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3" (UID: "b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:29:31.264669 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.264607 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3-kube-api-access-9gsq6" pod "b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3" (UID: "b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3"). InnerVolumeSpecName "kube-api-access-9gsq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:29:31.362672 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.362636 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-5fc65f84db-2jcf4"] Nov 02 23:29:31.362907 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.362849 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-5fc65f84db-2jcf4" podUID="554c20cb-570b-49f7-af7a-b9eb11177564" containerName="payload-processing" containerID="cri-o://4fc82e223fe829fe5bca6b552c7882239df85fbf3cf56673ad0b0bdf7557fbac" gracePeriod=30 Nov 02 23:29:31.363900 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.363866 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:29:31.364047 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.363923 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gsq6\" (UniqueName: \"kubernetes.io/projected/b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3-kube-api-access-9gsq6\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:29:31.364047 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.363936 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:29:31.425197 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.425107 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-7cd9d669b7-92tmp"] Nov 02 23:29:31.425642 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.425360 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-7cd9d669b7-92tmp" podUID="cbb5f075-306d-4826-a938-5363979473cc" containerName="payload-pre-processing" containerID="cri-o://3330427ef8c4ab954b3693c211e9c988499ba4d05683287afcf2315cdf6fde66" gracePeriod=30 Nov 02 23:29:31.512237 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.512203 5076 generic.go:356] "Generic (PLEG): container finished" podID="554c20cb-570b-49f7-af7a-b9eb11177564" containerID="4fc82e223fe829fe5bca6b552c7882239df85fbf3cf56673ad0b0bdf7557fbac" exitCode=0 Nov 02 23:29:31.512388 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.512270 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-5fc65f84db-2jcf4" event={"ID":"554c20cb-570b-49f7-af7a-b9eb11177564","Type":"ContainerDied","Data":"4fc82e223fe829fe5bca6b552c7882239df85fbf3cf56673ad0b0bdf7557fbac"} Nov 02 23:29:31.513580 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.513556 5076 generic.go:356] "Generic (PLEG): container finished" podID="b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3" containerID="c28e40e0f1feefa98b64ec8581c52360f57c4732603a6cf596306a9e1cccba6f" exitCode=0 Nov 02 23:29:31.513694 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.513592 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps" event={"ID":"b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3","Type":"ContainerDied","Data":"c28e40e0f1feefa98b64ec8581c52360f57c4732603a6cf596306a9e1cccba6f"} Nov 02 23:29:31.513694 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.513618 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps" Nov 02 23:29:31.513694 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.513628 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps" event={"ID":"b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3","Type":"ContainerDied","Data":"b374cdc937eaefa90a5b8c8a508c6524ec0b89248b9e2fc9515a69a0bd66bc75"} Nov 02 23:29:31.513694 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.513648 5076 scope.go:122] "RemoveContainer" containerID="c28e40e0f1feefa98b64ec8581c52360f57c4732603a6cf596306a9e1cccba6f" Nov 02 23:29:31.539204 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.539178 5076 scope.go:122] "RemoveContainer" containerID="c28e40e0f1feefa98b64ec8581c52360f57c4732603a6cf596306a9e1cccba6f" Nov 02 23:29:31.539526 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:29:31.539500 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c28e40e0f1feefa98b64ec8581c52360f57c4732603a6cf596306a9e1cccba6f\": container with ID starting with c28e40e0f1feefa98b64ec8581c52360f57c4732603a6cf596306a9e1cccba6f not found: ID does not exist" containerID="c28e40e0f1feefa98b64ec8581c52360f57c4732603a6cf596306a9e1cccba6f" Nov 02 23:29:31.539642 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.539539 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c28e40e0f1feefa98b64ec8581c52360f57c4732603a6cf596306a9e1cccba6f"} err="failed to get container status \"c28e40e0f1feefa98b64ec8581c52360f57c4732603a6cf596306a9e1cccba6f\": rpc error: code = NotFound desc = could not find container \"c28e40e0f1feefa98b64ec8581c52360f57c4732603a6cf596306a9e1cccba6f\": container with ID starting with c28e40e0f1feefa98b64ec8581c52360f57c4732603a6cf596306a9e1cccba6f not found: ID does not exist" Nov 02 23:29:31.705832 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.705790 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps"] Nov 02 23:29:31.708418 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.708391 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fdfca-876f4c9d7-stfps"] Nov 02 23:29:31.745818 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.745793 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-5fc65f84db-2jcf4" Nov 02 23:29:31.750121 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.750099 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-7cd9d669b7-92tmp" Nov 02 23:29:31.870571 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.870541 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/554c20cb-570b-49f7-af7a-b9eb11177564-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/554c20cb-570b-49f7-af7a-b9eb11177564-plugins-config-volume\") pod \"554c20cb-570b-49f7-af7a-b9eb11177564\" (UID: \"554c20cb-570b-49f7-af7a-b9eb11177564\") " Nov 02 23:29:31.870760 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.870586 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbb5f075-306d-4826-a938-5363979473cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbb5f075-306d-4826-a938-5363979473cc-plugins-config-volume\") pod \"cbb5f075-306d-4826-a938-5363979473cc\" (UID: \"cbb5f075-306d-4826-a938-5363979473cc\") " Nov 02 23:29:31.870760 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.870669 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbb5f075-306d-4826-a938-5363979473cc-kube-api-access-7wf9p\" (UniqueName: \"kubernetes.io/projected/cbb5f075-306d-4826-a938-5363979473cc-kube-api-access-7wf9p\") pod \"cbb5f075-306d-4826-a938-5363979473cc\" (UID: \"cbb5f075-306d-4826-a938-5363979473cc\") " Nov 02 23:29:31.870760 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.870740 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/554c20cb-570b-49f7-af7a-b9eb11177564-kube-api-access-lc2tb\" (UniqueName: \"kubernetes.io/projected/554c20cb-570b-49f7-af7a-b9eb11177564-kube-api-access-lc2tb\") pod \"554c20cb-570b-49f7-af7a-b9eb11177564\" (UID: \"554c20cb-570b-49f7-af7a-b9eb11177564\") " Nov 02 23:29:31.870987 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.870963 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/554c20cb-570b-49f7-af7a-b9eb11177564-plugins-config-volume" pod "554c20cb-570b-49f7-af7a-b9eb11177564" (UID: "554c20cb-570b-49f7-af7a-b9eb11177564"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:29:31.871050 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.871025 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbb5f075-306d-4826-a938-5363979473cc-plugins-config-volume" pod "cbb5f075-306d-4826-a938-5363979473cc" (UID: "cbb5f075-306d-4826-a938-5363979473cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:29:31.871216 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.871201 5076 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/554c20cb-570b-49f7-af7a-b9eb11177564-plugins-config-volume\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:29:31.871280 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.871221 5076 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbb5f075-306d-4826-a938-5363979473cc-plugins-config-volume\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:29:31.872924 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.872899 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/554c20cb-570b-49f7-af7a-b9eb11177564-kube-api-access-lc2tb" pod "554c20cb-570b-49f7-af7a-b9eb11177564" (UID: "554c20cb-570b-49f7-af7a-b9eb11177564"). InnerVolumeSpecName "kube-api-access-lc2tb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:29:31.873059 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.872983 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbb5f075-306d-4826-a938-5363979473cc-kube-api-access-7wf9p" pod "cbb5f075-306d-4826-a938-5363979473cc" (UID: "cbb5f075-306d-4826-a938-5363979473cc"). InnerVolumeSpecName "kube-api-access-7wf9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:29:31.974457 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.974399 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wf9p\" (UniqueName: \"kubernetes.io/projected/cbb5f075-306d-4826-a938-5363979473cc-kube-api-access-7wf9p\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:29:31.974457 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:31.974431 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc2tb\" (UniqueName: \"kubernetes.io/projected/554c20cb-570b-49f7-af7a-b9eb11177564-kube-api-access-lc2tb\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:29:32.520802 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:32.520769 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-5fc65f84db-2jcf4" Nov 02 23:29:32.520802 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:32.520794 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-5fc65f84db-2jcf4" event={"ID":"554c20cb-570b-49f7-af7a-b9eb11177564","Type":"ContainerDied","Data":"34e8b2f2c13f713be11d90520b81be3f30dd51a31f9d10e0f0f62868839a5fc5"} Nov 02 23:29:32.521317 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:32.520867 5076 scope.go:122] "RemoveContainer" containerID="4fc82e223fe829fe5bca6b552c7882239df85fbf3cf56673ad0b0bdf7557fbac" Nov 02 23:29:32.522173 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:32.522149 5076 generic.go:356] "Generic (PLEG): container finished" podID="cbb5f075-306d-4826-a938-5363979473cc" containerID="3330427ef8c4ab954b3693c211e9c988499ba4d05683287afcf2315cdf6fde66" exitCode=0 Nov 02 23:29:32.522239 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:32.522191 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-7cd9d669b7-92tmp" event={"ID":"cbb5f075-306d-4826-a938-5363979473cc","Type":"ContainerDied","Data":"3330427ef8c4ab954b3693c211e9c988499ba4d05683287afcf2315cdf6fde66"} Nov 02 23:29:32.522239 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:32.522195 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-7cd9d669b7-92tmp" Nov 02 23:29:32.522239 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:32.522220 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-7cd9d669b7-92tmp" event={"ID":"cbb5f075-306d-4826-a938-5363979473cc","Type":"ContainerDied","Data":"187f6bc7aa521ad7a1e6ecda10e3af4e4b078d00cdec04ced50941feb4e74c48"} Nov 02 23:29:32.530811 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:32.530792 5076 scope.go:122] "RemoveContainer" containerID="3330427ef8c4ab954b3693c211e9c988499ba4d05683287afcf2315cdf6fde66" Nov 02 23:29:32.540419 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:32.540404 5076 scope.go:122] "RemoveContainer" containerID="3330427ef8c4ab954b3693c211e9c988499ba4d05683287afcf2315cdf6fde66" Nov 02 23:29:32.540693 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:29:32.540663 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3330427ef8c4ab954b3693c211e9c988499ba4d05683287afcf2315cdf6fde66\": container with ID starting with 3330427ef8c4ab954b3693c211e9c988499ba4d05683287afcf2315cdf6fde66 not found: ID does not exist" containerID="3330427ef8c4ab954b3693c211e9c988499ba4d05683287afcf2315cdf6fde66" Nov 02 23:29:32.540747 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:32.540705 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3330427ef8c4ab954b3693c211e9c988499ba4d05683287afcf2315cdf6fde66"} err="failed to get container status \"3330427ef8c4ab954b3693c211e9c988499ba4d05683287afcf2315cdf6fde66\": rpc error: code = NotFound desc = could not find container \"3330427ef8c4ab954b3693c211e9c988499ba4d05683287afcf2315cdf6fde66\": container with ID starting with 3330427ef8c4ab954b3693c211e9c988499ba4d05683287afcf2315cdf6fde66 not found: ID does not exist" Nov 02 23:29:32.677639 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:32.677602 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-5fc65f84db-2jcf4"] Nov 02 23:29:32.681810 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:32.681782 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-5fc65f84db-2jcf4"] Nov 02 23:29:32.785518 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:32.785433 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-7cd9d669b7-92tmp"] Nov 02 23:29:32.790351 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:32.790324 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-7cd9d669b7-92tmp"] Nov 02 23:29:33.148823 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:33.148744 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="554c20cb-570b-49f7-af7a-b9eb11177564" path="/var/lib/kubelet/pods/554c20cb-570b-49f7-af7a-b9eb11177564/volumes" Nov 02 23:29:33.149280 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:33.149264 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3" path="/var/lib/kubelet/pods/b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3/volumes" Nov 02 23:29:33.149666 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:33.149652 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbb5f075-306d-4826-a938-5363979473cc" path="/var/lib/kubelet/pods/cbb5f075-306d-4826-a938-5363979473cc/volumes" Nov 02 23:29:35.544068 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:35.544032 5076 generic.go:356] "Generic (PLEG): container finished" podID="8c5807a1-409d-4188-abb7-1bf3bfb8e07d" containerID="c7800ec2dfb7b4ff4805a5b9a9a863134b456ee5f343daaea2c49ab10487e1e3" exitCode=0 Nov 02 23:29:35.544489 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:35.544101 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" event={"ID":"8c5807a1-409d-4188-abb7-1bf3bfb8e07d","Type":"ContainerDied","Data":"c7800ec2dfb7b4ff4805a5b9a9a863134b456ee5f343daaea2c49ab10487e1e3"} Nov 02 23:29:35.544489 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:35.544196 5076 scope.go:122] "RemoveContainer" containerID="bb3b65d54e0566441a9ab3fadbc4e9cd77ea6305086388174b1fd87a19c4fdf6" Nov 02 23:29:35.698457 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:35.698427 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" Nov 02 23:29:35.830937 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:35.830882 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-credential-socket\") pod \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\" (UID: \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\") " Nov 02 23:29:35.831114 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:35.830958 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-istio-podinfo\") pod \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\" (UID: \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\") " Nov 02 23:29:35.831114 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:35.831014 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-kube-api-access-w6xpv\" (UniqueName: \"kubernetes.io/projected/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-kube-api-access-w6xpv\") pod \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\" (UID: \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\") " Nov 02 23:29:35.831114 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:35.831074 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-istiod-ca-cert\") pod \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\" (UID: \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\") " Nov 02 23:29:35.831275 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:35.831145 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-workload-socket\") pod \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\" (UID: \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\") " Nov 02 23:29:35.831275 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:35.831213 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-credential-socket" pod "8c5807a1-409d-4188-abb7-1bf3bfb8e07d" (UID: "8c5807a1-409d-4188-abb7-1bf3bfb8e07d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:29:35.831275 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:35.831268 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-istio-token\" (UniqueName: \"kubernetes.io/projected/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-istio-token\") pod \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\" (UID: \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\") " Nov 02 23:29:35.831431 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:35.831317 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-istio-envoy\") pod \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\" (UID: \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\") " Nov 02 23:29:35.831497 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:35.831440 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-workload-socket" pod "8c5807a1-409d-4188-abb7-1bf3bfb8e07d" (UID: "8c5807a1-409d-4188-abb7-1bf3bfb8e07d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:29:35.831563 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:35.831521 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-istio-data\") pod \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\" (UID: \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\") " Nov 02 23:29:35.831563 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:35.831544 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-istiod-ca-cert" pod "8c5807a1-409d-4188-abb7-1bf3bfb8e07d" (UID: "8c5807a1-409d-4188-abb7-1bf3bfb8e07d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:29:35.831677 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:35.831574 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-workload-certs\") pod \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\" (UID: \"8c5807a1-409d-4188-abb7-1bf3bfb8e07d\") " Nov 02 23:29:35.831836 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:35.831817 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-istio-data" pod "8c5807a1-409d-4188-abb7-1bf3bfb8e07d" (UID: "8c5807a1-409d-4188-abb7-1bf3bfb8e07d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:29:35.832034 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:35.832015 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-workload-certs" pod "8c5807a1-409d-4188-abb7-1bf3bfb8e07d" (UID: "8c5807a1-409d-4188-abb7-1bf3bfb8e07d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:29:35.832106 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:35.832080 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:29:35.832160 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:35.832103 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:29:35.832160 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:35.832119 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:29:35.832160 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:35.832134 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:29:35.833838 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:35.833807 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-istio-token" pod "8c5807a1-409d-4188-abb7-1bf3bfb8e07d" (UID: "8c5807a1-409d-4188-abb7-1bf3bfb8e07d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:29:35.833838 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:35.833810 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-istio-podinfo" pod "8c5807a1-409d-4188-abb7-1bf3bfb8e07d" (UID: "8c5807a1-409d-4188-abb7-1bf3bfb8e07d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:29:35.834866 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:35.834842 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-kube-api-access-w6xpv" pod "8c5807a1-409d-4188-abb7-1bf3bfb8e07d" (UID: "8c5807a1-409d-4188-abb7-1bf3bfb8e07d"). InnerVolumeSpecName "kube-api-access-w6xpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:29:35.835107 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:35.835070 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-istio-envoy" pod "8c5807a1-409d-4188-abb7-1bf3bfb8e07d" (UID: "8c5807a1-409d-4188-abb7-1bf3bfb8e07d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:29:35.933865 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:35.933765 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:29:35.934051 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:35.933871 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:29:35.934051 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:35.933903 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6xpv\" (UniqueName: \"kubernetes.io/projected/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-kube-api-access-w6xpv\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:29:35.934051 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:35.933919 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:29:35.934051 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:35.933932 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c5807a1-409d-4188-abb7-1bf3bfb8e07d-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:29:36.552084 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:36.552055 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" Nov 02 23:29:36.552607 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:36.552097 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4" event={"ID":"8c5807a1-409d-4188-abb7-1bf3bfb8e07d","Type":"ContainerDied","Data":"b3101d3965b07a98b73d74f47e6426e6e4814118b9a885c984c3566eb5bee42e"} Nov 02 23:29:36.552607 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:36.552159 5076 scope.go:122] "RemoveContainer" containerID="c7800ec2dfb7b4ff4805a5b9a9a863134b456ee5f343daaea2c49ab10487e1e3" Nov 02 23:29:36.746117 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:36.746066 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9"] Nov 02 23:29:36.746359 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:36.746334 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" podUID="c3d41320-5841-4706-9e82-e5619986d711" containerName="istio-proxy" containerID="cri-o://3f5f1763c85a39a78f247fa13bde6c8cd5da438d76008537260d4b8c7ce94a13" gracePeriod=30 Nov 02 23:29:36.798562 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:36.798528 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4"] Nov 02 23:29:36.803057 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:36.802989 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-443fdf-openshift-default-59cb848dbf-z68k4"] Nov 02 23:29:37.148285 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:37.148206 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c5807a1-409d-4188-abb7-1bf3bfb8e07d" path="/var/lib/kubelet/pods/8c5807a1-409d-4188-abb7-1bf3bfb8e07d/volumes" Nov 02 23:29:38.684427 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.684389 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj"] Nov 02 23:29:38.685046 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.685025 5076 cpu_manager.go:414] "containerMap: removing container" podUID="cbb5f075-306d-4826-a938-5363979473cc" containerName="payload-pre-processing" Nov 02 23:29:38.685140 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.685048 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbb5f075-306d-4826-a938-5363979473cc" containerName="payload-pre-processing" Nov 02 23:29:38.685140 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.685064 5076 cpu_manager.go:414] "containerMap: removing container" podUID="554c20cb-570b-49f7-af7a-b9eb11177564" containerName="payload-processing" Nov 02 23:29:38.685140 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.685071 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="554c20cb-570b-49f7-af7a-b9eb11177564" containerName="payload-processing" Nov 02 23:29:38.685140 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.685097 5076 cpu_manager.go:414] "containerMap: removing container" podUID="8c5807a1-409d-4188-abb7-1bf3bfb8e07d" containerName="istio-proxy" Nov 02 23:29:38.685140 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.685102 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c5807a1-409d-4188-abb7-1bf3bfb8e07d" containerName="istio-proxy" Nov 02 23:29:38.685140 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.685108 5076 cpu_manager.go:414] "containerMap: removing container" podUID="8c5807a1-409d-4188-abb7-1bf3bfb8e07d" containerName="istio-proxy" Nov 02 23:29:38.685140 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.685114 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c5807a1-409d-4188-abb7-1bf3bfb8e07d" containerName="istio-proxy" Nov 02 23:29:38.685140 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.685124 5076 cpu_manager.go:414] "containerMap: removing container" podUID="b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3" containerName="maas-api" Nov 02 23:29:38.685140 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.685129 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3" containerName="maas-api" Nov 02 23:29:38.685475 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.685237 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbb5f075-306d-4826-a938-5363979473cc" containerName="payload-pre-processing" Nov 02 23:29:38.685475 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.685248 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="554c20cb-570b-49f7-af7a-b9eb11177564" containerName="payload-processing" Nov 02 23:29:38.685475 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.685259 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c5807a1-409d-4188-abb7-1bf3bfb8e07d" containerName="istio-proxy" Nov 02 23:29:38.685475 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.685264 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c5807a1-409d-4188-abb7-1bf3bfb8e07d" containerName="istio-proxy" Nov 02 23:29:38.685475 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.685269 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="b615d17d-c437-4fe9-8e69-3fdbcfa0a1b3" containerName="maas-api" Nov 02 23:29:38.689836 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.689815 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" Nov 02 23:29:38.765551 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.764732 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8d06dd9-47b5-442a-ad96-f8616e6570a8-istiod-ca-cert\") pod \"e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj\" (UID: \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\") " pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" Nov 02 23:29:38.765551 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.764807 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8d06dd9-47b5-442a-ad96-f8616e6570a8-credential-socket\") pod \"e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj\" (UID: \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\") " pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" Nov 02 23:29:38.765551 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.764872 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txjw8\" (UniqueName: \"kubernetes.io/projected/e8d06dd9-47b5-442a-ad96-f8616e6570a8-kube-api-access-txjw8\") pod \"e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj\" (UID: \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\") " pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" Nov 02 23:29:38.765551 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.764924 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8d06dd9-47b5-442a-ad96-f8616e6570a8-istio-token\") pod \"e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj\" (UID: \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\") " pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" Nov 02 23:29:38.765551 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.764963 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8d06dd9-47b5-442a-ad96-f8616e6570a8-istio-data\") pod \"e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj\" (UID: \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\") " pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" Nov 02 23:29:38.765551 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.764988 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8d06dd9-47b5-442a-ad96-f8616e6570a8-istio-envoy\") pod \"e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj\" (UID: \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\") " pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" Nov 02 23:29:38.765551 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.765034 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8d06dd9-47b5-442a-ad96-f8616e6570a8-workload-certs\") pod \"e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj\" (UID: \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\") " pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" Nov 02 23:29:38.765551 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.765059 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8d06dd9-47b5-442a-ad96-f8616e6570a8-istio-podinfo\") pod \"e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj\" (UID: \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\") " pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" Nov 02 23:29:38.765551 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.765202 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8d06dd9-47b5-442a-ad96-f8616e6570a8-workload-socket\") pod \"e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj\" (UID: \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\") " pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" Nov 02 23:29:38.788143 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.788108 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-fb5fa2-openshift-default-dockercfg-vd9k2\"" Nov 02 23:29:38.808478 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.808023 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj"] Nov 02 23:29:38.875314 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.867384 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8d06dd9-47b5-442a-ad96-f8616e6570a8-istio-token\") pod \"e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj\" (UID: \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\") " pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" Nov 02 23:29:38.875314 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.867438 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8d06dd9-47b5-442a-ad96-f8616e6570a8-istio-data\") pod \"e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj\" (UID: \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\") " pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" Nov 02 23:29:38.875314 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.867469 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8d06dd9-47b5-442a-ad96-f8616e6570a8-istio-envoy\") pod \"e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj\" (UID: \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\") " pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" Nov 02 23:29:38.875314 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.867514 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8d06dd9-47b5-442a-ad96-f8616e6570a8-workload-certs\") pod \"e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj\" (UID: \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\") " pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" Nov 02 23:29:38.875314 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.867546 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8d06dd9-47b5-442a-ad96-f8616e6570a8-istio-podinfo\") pod \"e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj\" (UID: \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\") " pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" Nov 02 23:29:38.875314 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.867673 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8d06dd9-47b5-442a-ad96-f8616e6570a8-workload-socket\") pod \"e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj\" (UID: \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\") " pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" Nov 02 23:29:38.875314 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.867738 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8d06dd9-47b5-442a-ad96-f8616e6570a8-istiod-ca-cert\") pod \"e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj\" (UID: \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\") " pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" Nov 02 23:29:38.875314 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.867794 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8d06dd9-47b5-442a-ad96-f8616e6570a8-credential-socket\") pod \"e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj\" (UID: \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\") " pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" Nov 02 23:29:38.875314 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.867861 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txjw8\" (UniqueName: \"kubernetes.io/projected/e8d06dd9-47b5-442a-ad96-f8616e6570a8-kube-api-access-txjw8\") pod \"e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj\" (UID: \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\") " pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" Nov 02 23:29:38.875314 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.868834 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8d06dd9-47b5-442a-ad96-f8616e6570a8-istio-data\") pod \"e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj\" (UID: \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\") " pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" Nov 02 23:29:38.875314 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.869985 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8d06dd9-47b5-442a-ad96-f8616e6570a8-workload-socket\") pod \"e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj\" (UID: \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\") " pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" Nov 02 23:29:38.875314 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.870338 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8d06dd9-47b5-442a-ad96-f8616e6570a8-workload-certs\") pod \"e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj\" (UID: \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\") " pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" Nov 02 23:29:38.875314 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.872300 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8d06dd9-47b5-442a-ad96-f8616e6570a8-istio-envoy\") pod \"e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj\" (UID: \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\") " pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" Nov 02 23:29:38.875314 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.872978 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8d06dd9-47b5-442a-ad96-f8616e6570a8-istiod-ca-cert\") pod \"e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj\" (UID: \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\") " pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" Nov 02 23:29:38.875314 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.873251 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8d06dd9-47b5-442a-ad96-f8616e6570a8-credential-socket\") pod \"e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj\" (UID: \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\") " pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" Nov 02 23:29:38.884920 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.884754 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8d06dd9-47b5-442a-ad96-f8616e6570a8-istio-podinfo\") pod \"e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj\" (UID: \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\") " pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" Nov 02 23:29:38.923326 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.923290 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8d06dd9-47b5-442a-ad96-f8616e6570a8-istio-token\") pod \"e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj\" (UID: \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\") " pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" Nov 02 23:29:38.923467 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:38.923349 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txjw8\" (UniqueName: \"kubernetes.io/projected/e8d06dd9-47b5-442a-ad96-f8616e6570a8-kube-api-access-txjw8\") pod \"e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj\" (UID: \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\") " pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" Nov 02 23:29:39.002599 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:39.002520 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" Nov 02 23:29:39.422187 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:39.422160 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj"] Nov 02 23:29:39.423962 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:29:39.423932 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8d06dd9_47b5_442a_ad96_f8616e6570a8.slice/crio-e905672868ea0e82b42082637973b2229f6f78d8ab23848a83409800141276ee WatchSource:0}: Error finding container e905672868ea0e82b42082637973b2229f6f78d8ab23848a83409800141276ee: Status 404 returned error can't find the container with id e905672868ea0e82b42082637973b2229f6f78d8ab23848a83409800141276ee Nov 02 23:29:39.426399 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:39.426364 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:29:39.426504 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:39.426441 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:29:39.426504 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:39.426486 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:29:39.573697 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:39.573663 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" event={"ID":"e8d06dd9-47b5-442a-ad96-f8616e6570a8","Type":"ContainerStarted","Data":"f01d415c678dd1f861af35a82f6eb8c5b5bb378a8770c175d19b4360b25ff7d5"} Nov 02 23:29:39.573850 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:39.573713 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" event={"ID":"e8d06dd9-47b5-442a-ad96-f8616e6570a8","Type":"ContainerStarted","Data":"e905672868ea0e82b42082637973b2229f6f78d8ab23848a83409800141276ee"} Nov 02 23:29:39.687495 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:39.687442 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" podStartSLOduration=1.687425164 podStartE2EDuration="1.687425164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:29:39.68713716 +0000 UTC m=+2228.655668945" watchObservedRunningTime="2026-11-02 23:29:39.687425164 +0000 UTC m=+2228.655956948" Nov 02 23:29:39.886787 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:29:39.886741 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:29:40.003303 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:40.003214 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" Nov 02 23:29:40.009298 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:40.009271 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" Nov 02 23:29:40.579979 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:40.579946 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" Nov 02 23:29:40.580965 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:40.580945 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" Nov 02 23:29:42.018297 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.018234 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" Nov 02 23:29:42.128040 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.128007 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3d41320-5841-4706-9e82-e5619986d711-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d41320-5841-4706-9e82-e5619986d711-credential-socket\") pod \"c3d41320-5841-4706-9e82-e5619986d711\" (UID: \"c3d41320-5841-4706-9e82-e5619986d711\") " Nov 02 23:29:42.128230 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.128100 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3d41320-5841-4706-9e82-e5619986d711-istio-token\" (UniqueName: \"kubernetes.io/projected/c3d41320-5841-4706-9e82-e5619986d711-istio-token\") pod \"c3d41320-5841-4706-9e82-e5619986d711\" (UID: \"c3d41320-5841-4706-9e82-e5619986d711\") " Nov 02 23:29:42.128230 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.128131 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3d41320-5841-4706-9e82-e5619986d711-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3d41320-5841-4706-9e82-e5619986d711-istiod-ca-cert\") pod \"c3d41320-5841-4706-9e82-e5619986d711\" (UID: \"c3d41320-5841-4706-9e82-e5619986d711\") " Nov 02 23:29:42.128230 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.128159 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3d41320-5841-4706-9e82-e5619986d711-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d41320-5841-4706-9e82-e5619986d711-workload-socket\") pod \"c3d41320-5841-4706-9e82-e5619986d711\" (UID: \"c3d41320-5841-4706-9e82-e5619986d711\") " Nov 02 23:29:42.128230 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.128189 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3d41320-5841-4706-9e82-e5619986d711-kube-api-access-88hgc\" (UniqueName: \"kubernetes.io/projected/c3d41320-5841-4706-9e82-e5619986d711-kube-api-access-88hgc\") pod \"c3d41320-5841-4706-9e82-e5619986d711\" (UID: \"c3d41320-5841-4706-9e82-e5619986d711\") " Nov 02 23:29:42.128230 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.128222 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3d41320-5841-4706-9e82-e5619986d711-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3d41320-5841-4706-9e82-e5619986d711-istio-podinfo\") pod \"c3d41320-5841-4706-9e82-e5619986d711\" (UID: \"c3d41320-5841-4706-9e82-e5619986d711\") " Nov 02 23:29:42.128466 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.128252 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3d41320-5841-4706-9e82-e5619986d711-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3d41320-5841-4706-9e82-e5619986d711-istio-envoy\") pod \"c3d41320-5841-4706-9e82-e5619986d711\" (UID: \"c3d41320-5841-4706-9e82-e5619986d711\") " Nov 02 23:29:42.128466 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.128283 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3d41320-5841-4706-9e82-e5619986d711-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3d41320-5841-4706-9e82-e5619986d711-workload-certs\") pod \"c3d41320-5841-4706-9e82-e5619986d711\" (UID: \"c3d41320-5841-4706-9e82-e5619986d711\") " Nov 02 23:29:42.128466 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.128372 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d41320-5841-4706-9e82-e5619986d711-credential-socket" pod "c3d41320-5841-4706-9e82-e5619986d711" (UID: "c3d41320-5841-4706-9e82-e5619986d711"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:29:42.128466 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.128408 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3d41320-5841-4706-9e82-e5619986d711-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3d41320-5841-4706-9e82-e5619986d711-istio-data\") pod \"c3d41320-5841-4706-9e82-e5619986d711\" (UID: \"c3d41320-5841-4706-9e82-e5619986d711\") " Nov 02 23:29:42.128678 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.128584 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3d41320-5841-4706-9e82-e5619986d711-istiod-ca-cert" pod "c3d41320-5841-4706-9e82-e5619986d711" (UID: "c3d41320-5841-4706-9e82-e5619986d711"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:29:42.128678 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.128612 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d41320-5841-4706-9e82-e5619986d711-workload-socket" pod "c3d41320-5841-4706-9e82-e5619986d711" (UID: "c3d41320-5841-4706-9e82-e5619986d711"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:29:42.128678 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.128646 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d41320-5841-4706-9e82-e5619986d711-workload-certs" pod "c3d41320-5841-4706-9e82-e5619986d711" (UID: "c3d41320-5841-4706-9e82-e5619986d711"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:29:42.128830 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.128786 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d41320-5841-4706-9e82-e5619986d711-istio-data" pod "c3d41320-5841-4706-9e82-e5619986d711" (UID: "c3d41320-5841-4706-9e82-e5619986d711"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:29:42.129073 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.129050 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d41320-5841-4706-9e82-e5619986d711-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:29:42.129192 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.129078 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3d41320-5841-4706-9e82-e5619986d711-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:29:42.129192 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.129093 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d41320-5841-4706-9e82-e5619986d711-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:29:42.129192 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.129108 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3d41320-5841-4706-9e82-e5619986d711-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:29:42.129192 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.129161 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3d41320-5841-4706-9e82-e5619986d711-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:29:42.130616 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.130593 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3d41320-5841-4706-9e82-e5619986d711-istio-podinfo" pod "c3d41320-5841-4706-9e82-e5619986d711" (UID: "c3d41320-5841-4706-9e82-e5619986d711"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:29:42.130616 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.130615 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3d41320-5841-4706-9e82-e5619986d711-istio-token" pod "c3d41320-5841-4706-9e82-e5619986d711" (UID: "c3d41320-5841-4706-9e82-e5619986d711"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:29:42.130763 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.130596 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3d41320-5841-4706-9e82-e5619986d711-kube-api-access-88hgc" pod "c3d41320-5841-4706-9e82-e5619986d711" (UID: "c3d41320-5841-4706-9e82-e5619986d711"). InnerVolumeSpecName "kube-api-access-88hgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:29:42.130763 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.130673 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d41320-5841-4706-9e82-e5619986d711-istio-envoy" pod "c3d41320-5841-4706-9e82-e5619986d711" (UID: "c3d41320-5841-4706-9e82-e5619986d711"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:29:42.230139 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.230106 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3d41320-5841-4706-9e82-e5619986d711-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:29:42.230139 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.230135 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88hgc\" (UniqueName: \"kubernetes.io/projected/c3d41320-5841-4706-9e82-e5619986d711-kube-api-access-88hgc\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:29:42.230139 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.230147 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3d41320-5841-4706-9e82-e5619986d711-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:29:42.230378 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.230156 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3d41320-5841-4706-9e82-e5619986d711-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:29:42.594369 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.594330 5076 generic.go:356] "Generic (PLEG): container finished" podID="c3d41320-5841-4706-9e82-e5619986d711" containerID="3f5f1763c85a39a78f247fa13bde6c8cd5da438d76008537260d4b8c7ce94a13" exitCode=0 Nov 02 23:29:42.594550 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.594383 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" Nov 02 23:29:42.594550 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.594408 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" event={"ID":"c3d41320-5841-4706-9e82-e5619986d711","Type":"ContainerDied","Data":"3f5f1763c85a39a78f247fa13bde6c8cd5da438d76008537260d4b8c7ce94a13"} Nov 02 23:29:42.594550 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.594462 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9" event={"ID":"c3d41320-5841-4706-9e82-e5619986d711","Type":"ContainerDied","Data":"7eb4ee84a1a19e332e74e9461a7d8434c646d4bb90b65ad2a8bb515577c91750"} Nov 02 23:29:42.594550 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.594486 5076 scope.go:122] "RemoveContainer" containerID="3f5f1763c85a39a78f247fa13bde6c8cd5da438d76008537260d4b8c7ce94a13" Nov 02 23:29:42.605340 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.605320 5076 scope.go:122] "RemoveContainer" containerID="0ef99bfbdc0426efb15add5364e6a19faa32f368b40c8e6e1edef5523018c00e" Nov 02 23:29:42.614602 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.614582 5076 scope.go:122] "RemoveContainer" containerID="3f5f1763c85a39a78f247fa13bde6c8cd5da438d76008537260d4b8c7ce94a13" Nov 02 23:29:42.614883 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:29:42.614865 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f5f1763c85a39a78f247fa13bde6c8cd5da438d76008537260d4b8c7ce94a13\": container with ID starting with 3f5f1763c85a39a78f247fa13bde6c8cd5da438d76008537260d4b8c7ce94a13 not found: ID does not exist" containerID="3f5f1763c85a39a78f247fa13bde6c8cd5da438d76008537260d4b8c7ce94a13" Nov 02 23:29:42.614955 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.614922 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f5f1763c85a39a78f247fa13bde6c8cd5da438d76008537260d4b8c7ce94a13"} err="failed to get container status \"3f5f1763c85a39a78f247fa13bde6c8cd5da438d76008537260d4b8c7ce94a13\": rpc error: code = NotFound desc = could not find container \"3f5f1763c85a39a78f247fa13bde6c8cd5da438d76008537260d4b8c7ce94a13\": container with ID starting with 3f5f1763c85a39a78f247fa13bde6c8cd5da438d76008537260d4b8c7ce94a13 not found: ID does not exist" Nov 02 23:29:42.614955 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.614943 5076 scope.go:122] "RemoveContainer" containerID="0ef99bfbdc0426efb15add5364e6a19faa32f368b40c8e6e1edef5523018c00e" Nov 02 23:29:42.615190 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:29:42.615168 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ef99bfbdc0426efb15add5364e6a19faa32f368b40c8e6e1edef5523018c00e\": container with ID starting with 0ef99bfbdc0426efb15add5364e6a19faa32f368b40c8e6e1edef5523018c00e not found: ID does not exist" containerID="0ef99bfbdc0426efb15add5364e6a19faa32f368b40c8e6e1edef5523018c00e" Nov 02 23:29:42.615250 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.615199 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ef99bfbdc0426efb15add5364e6a19faa32f368b40c8e6e1edef5523018c00e"} err="failed to get container status \"0ef99bfbdc0426efb15add5364e6a19faa32f368b40c8e6e1edef5523018c00e\": rpc error: code = NotFound desc = could not find container \"0ef99bfbdc0426efb15add5364e6a19faa32f368b40c8e6e1edef5523018c00e\": container with ID starting with 0ef99bfbdc0426efb15add5364e6a19faa32f368b40c8e6e1edef5523018c00e not found: ID does not exist" Nov 02 23:29:42.752186 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.752143 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9"] Nov 02 23:29:42.801013 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:42.800979 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-7fdfca-openshift-default-5d8466499f-6gzz9"] Nov 02 23:29:43.152594 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:43.152556 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3d41320-5841-4706-9e82-e5619986d711" path="/var/lib/kubelet/pods/c3d41320-5841-4706-9e82-e5619986d711/volumes" Nov 02 23:29:55.700178 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:55.700139 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9"] Nov 02 23:29:55.700674 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:55.700660 5076 cpu_manager.go:414] "containerMap: removing container" podUID="c3d41320-5841-4706-9e82-e5619986d711" containerName="istio-proxy" Nov 02 23:29:55.700674 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:55.700675 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3d41320-5841-4706-9e82-e5619986d711" containerName="istio-proxy" Nov 02 23:29:55.700783 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:55.700774 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3d41320-5841-4706-9e82-e5619986d711" containerName="istio-proxy" Nov 02 23:29:55.704044 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:55.704023 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9" Nov 02 23:29:55.798815 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:55.798774 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e8ca80b-ef9c-4321-a986-b832858f7fd4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9\" (UID: \"0e8ca80b-ef9c-4321-a986-b832858f7fd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9" Nov 02 23:29:55.799013 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:55.798971 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e8ca80b-ef9c-4321-a986-b832858f7fd4-metrics-certs\") pod \"maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9\" (UID: \"0e8ca80b-ef9c-4321-a986-b832858f7fd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9" Nov 02 23:29:55.799082 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:55.799016 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh8r6\" (UniqueName: \"kubernetes.io/projected/0e8ca80b-ef9c-4321-a986-b832858f7fd4-kube-api-access-dh8r6\") pod \"maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9\" (UID: \"0e8ca80b-ef9c-4321-a986-b832858f7fd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9" Nov 02 23:29:55.799230 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:55.799208 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e8ca80b-ef9c-4321-a986-b832858f7fd4-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9\" (UID: \"0e8ca80b-ef9c-4321-a986-b832858f7fd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9" Nov 02 23:29:55.801378 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:55.801356 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-02cdd9-serving-cert\"" Nov 02 23:29:55.804122 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:55.804102 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9"] Nov 02 23:29:55.900403 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:55.900371 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e8ca80b-ef9c-4321-a986-b832858f7fd4-metrics-certs\") pod \"maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9\" (UID: \"0e8ca80b-ef9c-4321-a986-b832858f7fd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9" Nov 02 23:29:55.900613 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:55.900420 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh8r6\" (UniqueName: \"kubernetes.io/projected/0e8ca80b-ef9c-4321-a986-b832858f7fd4-kube-api-access-dh8r6\") pod \"maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9\" (UID: \"0e8ca80b-ef9c-4321-a986-b832858f7fd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9" Nov 02 23:29:55.900613 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:55.900485 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e8ca80b-ef9c-4321-a986-b832858f7fd4-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9\" (UID: \"0e8ca80b-ef9c-4321-a986-b832858f7fd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9" Nov 02 23:29:55.900613 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:55.900524 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e8ca80b-ef9c-4321-a986-b832858f7fd4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9\" (UID: \"0e8ca80b-ef9c-4321-a986-b832858f7fd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9" Nov 02 23:29:55.900801 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:29:55.900661 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-02cdd9-serving-cert: secret "maas-api-e2e-praxis-b-02cdd9-serving-cert" not found Nov 02 23:29:55.900801 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:29:55.900723 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e8ca80b-ef9c-4321-a986-b832858f7fd4-maas-api-tls podName:0e8ca80b-ef9c-4321-a986-b832858f7fd4 nodeName:}" failed. No retries permitted until 2026-11-02 23:29:56.400706594 +0000 UTC m=+2245.369238370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0e8ca80b-ef9c-4321-a986-b832858f7fd4-maas-api-tls") pod "maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9" (UID: "0e8ca80b-ef9c-4321-a986-b832858f7fd4") : secret "maas-api-e2e-praxis-b-02cdd9-serving-cert" not found Nov 02 23:29:55.901182 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:55.901161 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e8ca80b-ef9c-4321-a986-b832858f7fd4-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9\" (UID: \"0e8ca80b-ef9c-4321-a986-b832858f7fd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9" Nov 02 23:29:55.902986 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:55.902965 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e8ca80b-ef9c-4321-a986-b832858f7fd4-metrics-certs\") pod \"maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9\" (UID: \"0e8ca80b-ef9c-4321-a986-b832858f7fd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9" Nov 02 23:29:55.972296 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:55.972217 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh8r6\" (UniqueName: \"kubernetes.io/projected/0e8ca80b-ef9c-4321-a986-b832858f7fd4-kube-api-access-dh8r6\") pod \"maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9\" (UID: \"0e8ca80b-ef9c-4321-a986-b832858f7fd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9" Nov 02 23:29:56.407968 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:56.407854 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e8ca80b-ef9c-4321-a986-b832858f7fd4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9\" (UID: \"0e8ca80b-ef9c-4321-a986-b832858f7fd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9" Nov 02 23:29:56.410335 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:56.410307 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e8ca80b-ef9c-4321-a986-b832858f7fd4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9\" (UID: \"0e8ca80b-ef9c-4321-a986-b832858f7fd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9" Nov 02 23:29:56.616974 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:56.616934 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9" Nov 02 23:29:56.985572 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:56.985543 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9"] Nov 02 23:29:56.987222 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:29:56.987185 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e8ca80b_ef9c_4321_a986_b832858f7fd4.slice/crio-36fab331de035d9aa0dbe8e85bec0511f129f19ba2c09bdd6fe1b04751cc1202 WatchSource:0}: Error finding container 36fab331de035d9aa0dbe8e85bec0511f129f19ba2c09bdd6fe1b04751cc1202: Status 404 returned error can't find the container with id 36fab331de035d9aa0dbe8e85bec0511f129f19ba2c09bdd6fe1b04751cc1202 Nov 02 23:29:56.988760 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:56.988741 5076 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 23:29:57.688520 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:57.688483 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9" event={"ID":"0e8ca80b-ef9c-4321-a986-b832858f7fd4","Type":"ContainerStarted","Data":"36fab331de035d9aa0dbe8e85bec0511f129f19ba2c09bdd6fe1b04751cc1202"} Nov 02 23:29:58.696716 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:58.696680 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9" event={"ID":"0e8ca80b-ef9c-4321-a986-b832858f7fd4","Type":"ContainerStarted","Data":"544680aacea4ae58955c718ed54b6184ea3ca2bd4f62d6c4cc0969c6234dcad7"} Nov 02 23:29:58.697152 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:58.696807 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9" Nov 02 23:29:58.807937 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:29:58.807869 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9" podStartSLOduration=2.965338223 podStartE2EDuration="3.80785441s" totalImagesPullingTime="842.516187ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:29:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:29:58.80565124 +0000 UTC m=+2247.774183025" watchObservedRunningTime="2026-11-02 23:29:58.80785441 +0000 UTC m=+2247.776386193" Nov 02 23:30:00.186826 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:00.186793 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-981724-29894370-gg89l"] Nov 02 23:30:00.187583 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:00.187560 5076 cpu_manager.go:414] "containerMap: removing container" podUID="c3d41320-5841-4706-9e82-e5619986d711" containerName="istio-proxy" Nov 02 23:30:00.187583 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:00.187584 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3d41320-5841-4706-9e82-e5619986d711" containerName="istio-proxy" Nov 02 23:30:00.187762 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:00.187722 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3d41320-5841-4706-9e82-e5619986d711" containerName="istio-proxy" Nov 02 23:30:00.191360 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:00.191334 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894370-dvc5d"] Nov 02 23:30:00.191476 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:00.191450 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-981724-29894370-gg89l" Nov 02 23:30:00.195825 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:00.195799 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894370-dvc5d" Nov 02 23:30:00.264274 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:00.264234 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k92gl\" (UniqueName: \"kubernetes.io/projected/35325f99-3e6a-4543-a120-d72d573557c8-kube-api-access-k92gl\") pod \"maas-api-key-cleanup-e2e-worker-w3-14b321-29894370-dvc5d\" (UID: \"35325f99-3e6a-4543-a120-d72d573557c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894370-dvc5d" Nov 02 23:30:00.264483 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:00.264458 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35325f99-3e6a-4543-a120-d72d573557c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-14b321-29894370-dvc5d\" (UID: \"35325f99-3e6a-4543-a120-d72d573557c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894370-dvc5d" Nov 02 23:30:00.264593 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:00.264571 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjglk\" (UniqueName: \"kubernetes.io/projected/cabbd992-6b17-4d40-9850-365a42e1335a-kube-api-access-sjglk\") pod \"maas-api-key-cleanup-e2e-ipp-a-981724-29894370-gg89l\" (UID: \"cabbd992-6b17-4d40-9850-365a42e1335a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-981724-29894370-gg89l" Nov 02 23:30:00.264654 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:00.264640 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cabbd992-6b17-4d40-9850-365a42e1335a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-981724-29894370-gg89l\" (UID: \"cabbd992-6b17-4d40-9850-365a42e1335a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-981724-29894370-gg89l" Nov 02 23:30:00.303722 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:00.303678 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-981724-29894370-gg89l"] Nov 02 23:30:00.309973 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:00.309942 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894370-dvc5d"] Nov 02 23:30:00.352803 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:00.352761 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:30:00.365943 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:00.365883 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35325f99-3e6a-4543-a120-d72d573557c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-14b321-29894370-dvc5d\" (UID: \"35325f99-3e6a-4543-a120-d72d573557c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894370-dvc5d" Nov 02 23:30:00.366126 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:00.366003 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjglk\" (UniqueName: \"kubernetes.io/projected/cabbd992-6b17-4d40-9850-365a42e1335a-kube-api-access-sjglk\") pod \"maas-api-key-cleanup-e2e-ipp-a-981724-29894370-gg89l\" (UID: \"cabbd992-6b17-4d40-9850-365a42e1335a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-981724-29894370-gg89l" Nov 02 23:30:00.366126 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:00.366063 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cabbd992-6b17-4d40-9850-365a42e1335a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-981724-29894370-gg89l\" (UID: \"cabbd992-6b17-4d40-9850-365a42e1335a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-981724-29894370-gg89l" Nov 02 23:30:00.366325 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:00.366171 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k92gl\" (UniqueName: \"kubernetes.io/projected/35325f99-3e6a-4543-a120-d72d573557c8-kube-api-access-k92gl\") pod \"maas-api-key-cleanup-e2e-worker-w3-14b321-29894370-dvc5d\" (UID: \"35325f99-3e6a-4543-a120-d72d573557c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894370-dvc5d" Nov 02 23:30:00.366695 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:00.366671 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35325f99-3e6a-4543-a120-d72d573557c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-14b321-29894370-dvc5d\" (UID: \"35325f99-3e6a-4543-a120-d72d573557c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894370-dvc5d" Nov 02 23:30:00.366819 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:00.366798 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cabbd992-6b17-4d40-9850-365a42e1335a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-981724-29894370-gg89l\" (UID: \"cabbd992-6b17-4d40-9850-365a42e1335a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-981724-29894370-gg89l" Nov 02 23:30:00.420459 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:00.420424 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k92gl\" (UniqueName: \"kubernetes.io/projected/35325f99-3e6a-4543-a120-d72d573557c8-kube-api-access-k92gl\") pod \"maas-api-key-cleanup-e2e-worker-w3-14b321-29894370-dvc5d\" (UID: \"35325f99-3e6a-4543-a120-d72d573557c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894370-dvc5d" Nov 02 23:30:00.420599 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:00.420489 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjglk\" (UniqueName: \"kubernetes.io/projected/cabbd992-6b17-4d40-9850-365a42e1335a-kube-api-access-sjglk\") pod \"maas-api-key-cleanup-e2e-ipp-a-981724-29894370-gg89l\" (UID: \"cabbd992-6b17-4d40-9850-365a42e1335a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-981724-29894370-gg89l" Nov 02 23:30:00.506981 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:00.506869 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-981724-29894370-gg89l" Nov 02 23:30:00.513260 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:00.513232 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894370-dvc5d" Nov 02 23:30:00.874428 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:00.874398 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-981724-29894370-gg89l"] Nov 02 23:30:00.875157 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:30:00.875115 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcabbd992_6b17_4d40_9850_365a42e1335a.slice/crio-49c2f4011afdf5ef4eb052b4c24ed20a657850239a181770a1c735c1a2084ddc WatchSource:0}: Error finding container 49c2f4011afdf5ef4eb052b4c24ed20a657850239a181770a1c735c1a2084ddc: Status 404 returned error can't find the container with id 49c2f4011afdf5ef4eb052b4c24ed20a657850239a181770a1c735c1a2084ddc Nov 02 23:30:00.922254 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:00.922229 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894370-dvc5d"] Nov 02 23:30:00.923286 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:30:00.923256 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35325f99_3e6a_4543_a120_d72d573557c8.slice/crio-1b7daa1d445d0daab4052e41cb943b1290f7d5afde73432fd28007e46549a409 WatchSource:0}: Error finding container 1b7daa1d445d0daab4052e41cb943b1290f7d5afde73432fd28007e46549a409: Status 404 returned error can't find the container with id 1b7daa1d445d0daab4052e41cb943b1290f7d5afde73432fd28007e46549a409 Nov 02 23:30:01.718274 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:01.718241 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894370-dvc5d" event={"ID":"35325f99-3e6a-4543-a120-d72d573557c8","Type":"ContainerStarted","Data":"1b6f6ad906a366ef09ac165011861236e5809b8a81f5a49b997a6b2f7535bf5c"} Nov 02 23:30:01.718668 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:01.718296 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894370-dvc5d" event={"ID":"35325f99-3e6a-4543-a120-d72d573557c8","Type":"ContainerStarted","Data":"1b7daa1d445d0daab4052e41cb943b1290f7d5afde73432fd28007e46549a409"} Nov 02 23:30:01.718668 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:01.718568 5076 scope.go:122] "RemoveContainer" containerID="1b6f6ad906a366ef09ac165011861236e5809b8a81f5a49b997a6b2f7535bf5c" Nov 02 23:30:01.720291 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:01.720197 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-981724-29894370-gg89l" event={"ID":"cabbd992-6b17-4d40-9850-365a42e1335a","Type":"ContainerStarted","Data":"f31839948ca4ef4ff473e854bd6cfafc2172870378aac4536f69463836494ae2"} Nov 02 23:30:01.720291 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:01.720257 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-981724-29894370-gg89l" event={"ID":"cabbd992-6b17-4d40-9850-365a42e1335a","Type":"ContainerStarted","Data":"49c2f4011afdf5ef4eb052b4c24ed20a657850239a181770a1c735c1a2084ddc"} Nov 02 23:30:01.720531 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:01.720435 5076 scope.go:122] "RemoveContainer" containerID="f31839948ca4ef4ff473e854bd6cfafc2172870378aac4536f69463836494ae2" Nov 02 23:30:02.729686 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:02.729647 5076 generic.go:356] "Generic (PLEG): container finished" podID="35325f99-3e6a-4543-a120-d72d573557c8" containerID="1b6f6ad906a366ef09ac165011861236e5809b8a81f5a49b997a6b2f7535bf5c" exitCode=60 Nov 02 23:30:02.729686 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:02.729683 5076 generic.go:356] "Generic (PLEG): container finished" podID="35325f99-3e6a-4543-a120-d72d573557c8" containerID="1711b8158070a16ee886078c75f6c573f565f42dcede46cfdf5cca0501243206" exitCode=60 Nov 02 23:30:02.730128 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:02.729783 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894370-dvc5d" event={"ID":"35325f99-3e6a-4543-a120-d72d573557c8","Type":"ContainerDied","Data":"1b6f6ad906a366ef09ac165011861236e5809b8a81f5a49b997a6b2f7535bf5c"} Nov 02 23:30:02.730128 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:02.729850 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894370-dvc5d" event={"ID":"35325f99-3e6a-4543-a120-d72d573557c8","Type":"ContainerDied","Data":"1711b8158070a16ee886078c75f6c573f565f42dcede46cfdf5cca0501243206"} Nov 02 23:30:02.730128 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:02.729881 5076 scope.go:122] "RemoveContainer" containerID="1b6f6ad906a366ef09ac165011861236e5809b8a81f5a49b997a6b2f7535bf5c" Nov 02 23:30:02.730128 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:02.730116 5076 scope.go:122] "RemoveContainer" containerID="1711b8158070a16ee886078c75f6c573f565f42dcede46cfdf5cca0501243206" Nov 02 23:30:02.730477 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:30:02.730452 5076 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-14b321-29894370-dvc5d_odh-ai-gateway-infra(35325f99-3e6a-4543-a120-d72d573557c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894370-dvc5d" podUID="35325f99-3e6a-4543-a120-d72d573557c8" Nov 02 23:30:02.731852 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:02.731830 5076 generic.go:356] "Generic (PLEG): container finished" podID="cabbd992-6b17-4d40-9850-365a42e1335a" containerID="f31839948ca4ef4ff473e854bd6cfafc2172870378aac4536f69463836494ae2" exitCode=60 Nov 02 23:30:02.731963 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:02.731853 5076 generic.go:356] "Generic (PLEG): container finished" podID="cabbd992-6b17-4d40-9850-365a42e1335a" containerID="bce98c52069ca3bfc72fbcd2d0a7c7073038aa878c303998bff8b8fffb5e588c" exitCode=60 Nov 02 23:30:02.731963 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:02.731932 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-981724-29894370-gg89l" event={"ID":"cabbd992-6b17-4d40-9850-365a42e1335a","Type":"ContainerDied","Data":"f31839948ca4ef4ff473e854bd6cfafc2172870378aac4536f69463836494ae2"} Nov 02 23:30:02.732059 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:02.731977 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-981724-29894370-gg89l" event={"ID":"cabbd992-6b17-4d40-9850-365a42e1335a","Type":"ContainerDied","Data":"bce98c52069ca3bfc72fbcd2d0a7c7073038aa878c303998bff8b8fffb5e588c"} Nov 02 23:30:02.732171 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:02.732158 5076 scope.go:122] "RemoveContainer" containerID="bce98c52069ca3bfc72fbcd2d0a7c7073038aa878c303998bff8b8fffb5e588c" Nov 02 23:30:02.732412 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:30:02.732394 5076 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-981724-29894370-gg89l_odh-ai-gateway-infra(cabbd992-6b17-4d40-9850-365a42e1335a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-981724-29894370-gg89l" podUID="cabbd992-6b17-4d40-9850-365a42e1335a" Nov 02 23:30:02.838346 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:02.838320 5076 scope.go:122] "RemoveContainer" containerID="1b6f6ad906a366ef09ac165011861236e5809b8a81f5a49b997a6b2f7535bf5c" Nov 02 23:30:02.838720 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:30:02.838688 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b6f6ad906a366ef09ac165011861236e5809b8a81f5a49b997a6b2f7535bf5c\": container with ID starting with 1b6f6ad906a366ef09ac165011861236e5809b8a81f5a49b997a6b2f7535bf5c not found: ID does not exist" containerID="1b6f6ad906a366ef09ac165011861236e5809b8a81f5a49b997a6b2f7535bf5c" Nov 02 23:30:02.838808 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:02.838728 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b6f6ad906a366ef09ac165011861236e5809b8a81f5a49b997a6b2f7535bf5c"} err="failed to get container status \"1b6f6ad906a366ef09ac165011861236e5809b8a81f5a49b997a6b2f7535bf5c\": rpc error: code = NotFound desc = could not find container \"1b6f6ad906a366ef09ac165011861236e5809b8a81f5a49b997a6b2f7535bf5c\": container with ID starting with 1b6f6ad906a366ef09ac165011861236e5809b8a81f5a49b997a6b2f7535bf5c not found: ID does not exist" Nov 02 23:30:02.838808 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:02.838748 5076 scope.go:122] "RemoveContainer" containerID="f31839948ca4ef4ff473e854bd6cfafc2172870378aac4536f69463836494ae2" Nov 02 23:30:02.854870 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:02.854848 5076 scope.go:122] "RemoveContainer" containerID="f31839948ca4ef4ff473e854bd6cfafc2172870378aac4536f69463836494ae2" Nov 02 23:30:02.855179 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:30:02.855162 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f31839948ca4ef4ff473e854bd6cfafc2172870378aac4536f69463836494ae2\": container with ID starting with f31839948ca4ef4ff473e854bd6cfafc2172870378aac4536f69463836494ae2 not found: ID does not exist" containerID="f31839948ca4ef4ff473e854bd6cfafc2172870378aac4536f69463836494ae2" Nov 02 23:30:02.855224 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:02.855187 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f31839948ca4ef4ff473e854bd6cfafc2172870378aac4536f69463836494ae2"} err="failed to get container status \"f31839948ca4ef4ff473e854bd6cfafc2172870378aac4536f69463836494ae2\": rpc error: code = NotFound desc = could not find container \"f31839948ca4ef4ff473e854bd6cfafc2172870378aac4536f69463836494ae2\": container with ID starting with f31839948ca4ef4ff473e854bd6cfafc2172870378aac4536f69463836494ae2 not found: ID does not exist" Nov 02 23:30:03.740115 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:03.740081 5076 scope.go:122] "RemoveContainer" containerID="1711b8158070a16ee886078c75f6c573f565f42dcede46cfdf5cca0501243206" Nov 02 23:30:03.740615 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:30:03.740428 5076 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-14b321-29894370-dvc5d_odh-ai-gateway-infra(35325f99-3e6a-4543-a120-d72d573557c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894370-dvc5d" podUID="35325f99-3e6a-4543-a120-d72d573557c8" Nov 02 23:30:03.745647 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:03.745625 5076 scope.go:122] "RemoveContainer" containerID="bce98c52069ca3bfc72fbcd2d0a7c7073038aa878c303998bff8b8fffb5e588c" Nov 02 23:30:03.745899 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:30:03.745868 5076 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-981724-29894370-gg89l_odh-ai-gateway-infra(cabbd992-6b17-4d40-9850-365a42e1335a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-981724-29894370-gg89l" podUID="cabbd992-6b17-4d40-9850-365a42e1335a" Nov 02 23:30:04.708796 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:04.708764 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9" Nov 02 23:30:09.882942 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:30:09.882877 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:30:17.141250 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:17.141219 5076 scope.go:122] "RemoveContainer" containerID="1711b8158070a16ee886078c75f6c573f565f42dcede46cfdf5cca0501243206" Nov 02 23:30:17.141735 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:17.141285 5076 scope.go:122] "RemoveContainer" containerID="bce98c52069ca3bfc72fbcd2d0a7c7073038aa878c303998bff8b8fffb5e588c" Nov 02 23:30:17.836311 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:17.836274 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894370-dvc5d" event={"ID":"35325f99-3e6a-4543-a120-d72d573557c8","Type":"ContainerStarted","Data":"67077a25b0fa3b507c93058702eb521953c9630911e57d4e2037b5babb6e10e1"} Nov 02 23:30:17.838476 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:17.838439 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-981724-29894370-gg89l" event={"ID":"cabbd992-6b17-4d40-9850-365a42e1335a","Type":"ContainerStarted","Data":"722a876e8969a5f31621bf5c834e0d598501d1a924c9f4745a1e2f6a6c0b943c"} Nov 02 23:30:17.945313 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:17.945266 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894370-dvc5d" podStartSLOduration=17.945251065 podStartE2EDuration="17.945251065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:30:17.944328649 +0000 UTC m=+2266.912860427" watchObservedRunningTime="2026-11-02 23:30:17.945251065 +0000 UTC m=+2266.913782849" Nov 02 23:30:18.053365 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:18.053322 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-981724-29894370-gg89l" podStartSLOduration=18.053308098 podStartE2EDuration="18.053308098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:30:18.051702374 +0000 UTC m=+2267.020234158" watchObservedRunningTime="2026-11-02 23:30:18.053308098 +0000 UTC m=+2267.021839882" Nov 02 23:30:18.847170 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:18.847140 5076 generic.go:356] "Generic (PLEG): container finished" podID="35325f99-3e6a-4543-a120-d72d573557c8" containerID="67077a25b0fa3b507c93058702eb521953c9630911e57d4e2037b5babb6e10e1" exitCode=60 Nov 02 23:30:18.847650 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:18.847208 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894370-dvc5d" event={"ID":"35325f99-3e6a-4543-a120-d72d573557c8","Type":"ContainerDied","Data":"67077a25b0fa3b507c93058702eb521953c9630911e57d4e2037b5babb6e10e1"} Nov 02 23:30:18.847650 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:18.847268 5076 scope.go:122] "RemoveContainer" containerID="1711b8158070a16ee886078c75f6c573f565f42dcede46cfdf5cca0501243206" Nov 02 23:30:18.847770 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:18.847686 5076 scope.go:122] "RemoveContainer" containerID="67077a25b0fa3b507c93058702eb521953c9630911e57d4e2037b5babb6e10e1" Nov 02 23:30:18.848089 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:30:18.848054 5076 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-14b321-29894370-dvc5d_odh-ai-gateway-infra(35325f99-3e6a-4543-a120-d72d573557c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894370-dvc5d" podUID="35325f99-3e6a-4543-a120-d72d573557c8" Nov 02 23:30:18.849423 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:18.849398 5076 generic.go:356] "Generic (PLEG): container finished" podID="cabbd992-6b17-4d40-9850-365a42e1335a" containerID="722a876e8969a5f31621bf5c834e0d598501d1a924c9f4745a1e2f6a6c0b943c" exitCode=60 Nov 02 23:30:18.849518 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:18.849472 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-981724-29894370-gg89l" event={"ID":"cabbd992-6b17-4d40-9850-365a42e1335a","Type":"ContainerDied","Data":"722a876e8969a5f31621bf5c834e0d598501d1a924c9f4745a1e2f6a6c0b943c"} Nov 02 23:30:18.849828 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:18.849813 5076 scope.go:122] "RemoveContainer" containerID="722a876e8969a5f31621bf5c834e0d598501d1a924c9f4745a1e2f6a6c0b943c" Nov 02 23:30:18.850096 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:30:18.850075 5076 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-981724-29894370-gg89l_odh-ai-gateway-infra(cabbd992-6b17-4d40-9850-365a42e1335a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-981724-29894370-gg89l" podUID="cabbd992-6b17-4d40-9850-365a42e1335a" Nov 02 23:30:18.859624 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:18.859605 5076 scope.go:122] "RemoveContainer" containerID="bce98c52069ca3bfc72fbcd2d0a7c7073038aa878c303998bff8b8fffb5e588c" Nov 02 23:30:18.954437 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:18.954404 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894370-dvc5d"] Nov 02 23:30:19.061605 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:19.061573 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-981724-29894370-gg89l"] Nov 02 23:30:20.103953 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:20.103924 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894370-dvc5d" Nov 02 23:30:20.114235 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:20.114206 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35325f99-3e6a-4543-a120-d72d573557c8-kube-api-access-k92gl\" (UniqueName: \"kubernetes.io/projected/35325f99-3e6a-4543-a120-d72d573557c8-kube-api-access-k92gl\") pod \"35325f99-3e6a-4543-a120-d72d573557c8\" (UID: \"35325f99-3e6a-4543-a120-d72d573557c8\") " Nov 02 23:30:20.114445 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:20.114423 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35325f99-3e6a-4543-a120-d72d573557c8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35325f99-3e6a-4543-a120-d72d573557c8-maas-api-service-ca\") pod \"35325f99-3e6a-4543-a120-d72d573557c8\" (UID: \"35325f99-3e6a-4543-a120-d72d573557c8\") " Nov 02 23:30:20.114843 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:20.114817 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35325f99-3e6a-4543-a120-d72d573557c8-maas-api-service-ca" pod "35325f99-3e6a-4543-a120-d72d573557c8" (UID: "35325f99-3e6a-4543-a120-d72d573557c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:30:20.116357 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:20.116330 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35325f99-3e6a-4543-a120-d72d573557c8-kube-api-access-k92gl" pod "35325f99-3e6a-4543-a120-d72d573557c8" (UID: "35325f99-3e6a-4543-a120-d72d573557c8"). InnerVolumeSpecName "kube-api-access-k92gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:30:20.139669 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:20.139648 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-981724-29894370-gg89l" Nov 02 23:30:20.215983 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:20.215935 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cabbd992-6b17-4d40-9850-365a42e1335a-kube-api-access-sjglk\" (UniqueName: \"kubernetes.io/projected/cabbd992-6b17-4d40-9850-365a42e1335a-kube-api-access-sjglk\") pod \"cabbd992-6b17-4d40-9850-365a42e1335a\" (UID: \"cabbd992-6b17-4d40-9850-365a42e1335a\") " Nov 02 23:30:20.216177 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:20.216059 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cabbd992-6b17-4d40-9850-365a42e1335a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cabbd992-6b17-4d40-9850-365a42e1335a-maas-api-service-ca\") pod \"cabbd992-6b17-4d40-9850-365a42e1335a\" (UID: \"cabbd992-6b17-4d40-9850-365a42e1335a\") " Nov 02 23:30:20.216383 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:20.216361 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k92gl\" (UniqueName: \"kubernetes.io/projected/35325f99-3e6a-4543-a120-d72d573557c8-kube-api-access-k92gl\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:30:20.216383 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:20.216384 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35325f99-3e6a-4543-a120-d72d573557c8-maas-api-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:30:20.216557 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:20.216383 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cabbd992-6b17-4d40-9850-365a42e1335a-maas-api-service-ca" pod "cabbd992-6b17-4d40-9850-365a42e1335a" (UID: "cabbd992-6b17-4d40-9850-365a42e1335a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:30:20.218137 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:20.218114 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cabbd992-6b17-4d40-9850-365a42e1335a-kube-api-access-sjglk" pod "cabbd992-6b17-4d40-9850-365a42e1335a" (UID: "cabbd992-6b17-4d40-9850-365a42e1335a"). InnerVolumeSpecName "kube-api-access-sjglk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:30:20.317375 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:20.317345 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjglk\" (UniqueName: \"kubernetes.io/projected/cabbd992-6b17-4d40-9850-365a42e1335a-kube-api-access-sjglk\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:30:20.317375 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:20.317372 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cabbd992-6b17-4d40-9850-365a42e1335a-maas-api-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:30:20.939948 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:20.939877 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894370-dvc5d" Nov 02 23:30:20.939948 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:20.939914 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894370-dvc5d" event={"ID":"35325f99-3e6a-4543-a120-d72d573557c8","Type":"ContainerDied","Data":"1b7daa1d445d0daab4052e41cb943b1290f7d5afde73432fd28007e46549a409"} Nov 02 23:30:20.940213 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:20.939988 5076 scope.go:122] "RemoveContainer" containerID="67077a25b0fa3b507c93058702eb521953c9630911e57d4e2037b5babb6e10e1" Nov 02 23:30:20.941562 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:20.941544 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-981724-29894370-gg89l" Nov 02 23:30:20.941660 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:20.941549 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-981724-29894370-gg89l" event={"ID":"cabbd992-6b17-4d40-9850-365a42e1335a","Type":"ContainerDied","Data":"49c2f4011afdf5ef4eb052b4c24ed20a657850239a181770a1c735c1a2084ddc"} Nov 02 23:30:20.959314 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:20.959290 5076 scope.go:122] "RemoveContainer" containerID="722a876e8969a5f31621bf5c834e0d598501d1a924c9f4745a1e2f6a6c0b943c" Nov 02 23:30:21.095880 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:21.095841 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894370-dvc5d"] Nov 02 23:30:21.099102 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:21.099076 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894370-dvc5d"] Nov 02 23:30:21.150020 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:21.149985 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35325f99-3e6a-4543-a120-d72d573557c8" path="/var/lib/kubelet/pods/35325f99-3e6a-4543-a120-d72d573557c8/volumes" Nov 02 23:30:21.201774 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:21.201687 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-981724-29894370-gg89l"] Nov 02 23:30:21.207206 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:21.207175 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-981724-29894370-gg89l"] Nov 02 23:30:23.147762 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:23.147730 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cabbd992-6b17-4d40-9850-365a42e1335a" path="/var/lib/kubelet/pods/cabbd992-6b17-4d40-9850-365a42e1335a/volumes" Nov 02 23:30:26.349227 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:26.349193 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-5fdb8d5c9f-8xdhn"] Nov 02 23:30:26.349742 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:26.349728 5076 cpu_manager.go:414] "containerMap: removing container" podUID="35325f99-3e6a-4543-a120-d72d573557c8" containerName="cleanup" Nov 02 23:30:26.349789 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:26.349743 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35325f99-3e6a-4543-a120-d72d573557c8" containerName="cleanup" Nov 02 23:30:26.349789 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:26.349759 5076 cpu_manager.go:414] "containerMap: removing container" podUID="cabbd992-6b17-4d40-9850-365a42e1335a" containerName="cleanup" Nov 02 23:30:26.349789 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:26.349764 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cabbd992-6b17-4d40-9850-365a42e1335a" containerName="cleanup" Nov 02 23:30:26.349789 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:26.349778 5076 cpu_manager.go:414] "containerMap: removing container" podUID="cabbd992-6b17-4d40-9850-365a42e1335a" containerName="cleanup" Nov 02 23:30:26.349789 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:26.349783 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cabbd992-6b17-4d40-9850-365a42e1335a" containerName="cleanup" Nov 02 23:30:26.349789 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:26.349790 5076 cpu_manager.go:414] "containerMap: removing container" podUID="cabbd992-6b17-4d40-9850-365a42e1335a" containerName="cleanup" Nov 02 23:30:26.349995 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:26.349795 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cabbd992-6b17-4d40-9850-365a42e1335a" containerName="cleanup" Nov 02 23:30:26.349995 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:26.349819 5076 cpu_manager.go:414] "containerMap: removing container" podUID="35325f99-3e6a-4543-a120-d72d573557c8" containerName="cleanup" Nov 02 23:30:26.349995 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:26.349823 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35325f99-3e6a-4543-a120-d72d573557c8" containerName="cleanup" Nov 02 23:30:26.349995 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:26.349911 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="35325f99-3e6a-4543-a120-d72d573557c8" containerName="cleanup" Nov 02 23:30:26.349995 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:26.349923 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="cabbd992-6b17-4d40-9850-365a42e1335a" containerName="cleanup" Nov 02 23:30:26.349995 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:26.349931 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="cabbd992-6b17-4d40-9850-365a42e1335a" containerName="cleanup" Nov 02 23:30:26.349995 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:26.349938 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="cabbd992-6b17-4d40-9850-365a42e1335a" containerName="cleanup" Nov 02 23:30:26.349995 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:26.349944 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="35325f99-3e6a-4543-a120-d72d573557c8" containerName="cleanup" Nov 02 23:30:26.353164 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:26.353141 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-5fdb8d5c9f-8xdhn" Nov 02 23:30:26.404940 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:26.404904 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6ef156b-92aa-4950-b83e-294e4adc2e77-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5ff336-5fdb8d5c9f-8xdhn\" (UID: \"a6ef156b-92aa-4950-b83e-294e4adc2e77\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-5fdb8d5c9f-8xdhn" Nov 02 23:30:26.405153 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:26.405128 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w47jg\" (UniqueName: \"kubernetes.io/projected/a6ef156b-92aa-4950-b83e-294e4adc2e77-kube-api-access-w47jg\") pod \"payload-pre-processing-e2e-ipp-b-5ff336-5fdb8d5c9f-8xdhn\" (UID: \"a6ef156b-92aa-4950-b83e-294e4adc2e77\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-5fdb8d5c9f-8xdhn" Nov 02 23:30:26.406310 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:26.406282 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-5fdb8d5c9f-8xdhn"] Nov 02 23:30:26.450248 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:26.450217 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5ff336\"" Nov 02 23:30:26.450761 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:26.450740 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5ff336-dockercfg-f2hfk\"" Nov 02 23:30:26.506746 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:26.506705 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w47jg\" (UniqueName: \"kubernetes.io/projected/a6ef156b-92aa-4950-b83e-294e4adc2e77-kube-api-access-w47jg\") pod \"payload-pre-processing-e2e-ipp-b-5ff336-5fdb8d5c9f-8xdhn\" (UID: \"a6ef156b-92aa-4950-b83e-294e4adc2e77\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-5fdb8d5c9f-8xdhn" Nov 02 23:30:26.507115 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:26.507090 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6ef156b-92aa-4950-b83e-294e4adc2e77-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5ff336-5fdb8d5c9f-8xdhn\" (UID: \"a6ef156b-92aa-4950-b83e-294e4adc2e77\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-5fdb8d5c9f-8xdhn" Nov 02 23:30:26.507822 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:26.507800 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6ef156b-92aa-4950-b83e-294e4adc2e77-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5ff336-5fdb8d5c9f-8xdhn\" (UID: \"a6ef156b-92aa-4950-b83e-294e4adc2e77\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-5fdb8d5c9f-8xdhn" Nov 02 23:30:26.561237 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:26.561207 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w47jg\" (UniqueName: \"kubernetes.io/projected/a6ef156b-92aa-4950-b83e-294e4adc2e77-kube-api-access-w47jg\") pod \"payload-pre-processing-e2e-ipp-b-5ff336-5fdb8d5c9f-8xdhn\" (UID: \"a6ef156b-92aa-4950-b83e-294e4adc2e77\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-5fdb8d5c9f-8xdhn" Nov 02 23:30:26.667488 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:26.667395 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-5fdb8d5c9f-8xdhn" Nov 02 23:30:27.036224 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:27.036185 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-5fdb8d5c9f-8xdhn"] Nov 02 23:30:27.037292 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:30:27.037263 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6ef156b_92aa_4950_b83e_294e4adc2e77.slice/crio-f03aa0a8e05df12b5cf7ecae8e87dbac7ecaf4332737c9a1ea75b4c8dd49eea6 WatchSource:0}: Error finding container f03aa0a8e05df12b5cf7ecae8e87dbac7ecaf4332737c9a1ea75b4c8dd49eea6: Status 404 returned error can't find the container with id f03aa0a8e05df12b5cf7ecae8e87dbac7ecaf4332737c9a1ea75b4c8dd49eea6 Nov 02 23:30:27.097112 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:27.097084 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5ff336-7d447bd878-v7jmb"] Nov 02 23:30:27.097696 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:27.097681 5076 cpu_manager.go:414] "containerMap: removing container" podUID="35325f99-3e6a-4543-a120-d72d573557c8" containerName="cleanup" Nov 02 23:30:27.097755 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:27.097698 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35325f99-3e6a-4543-a120-d72d573557c8" containerName="cleanup" Nov 02 23:30:27.097791 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:27.097781 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="35325f99-3e6a-4543-a120-d72d573557c8" containerName="cleanup" Nov 02 23:30:27.101270 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:27.101252 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5ff336-7d447bd878-v7jmb" Nov 02 23:30:27.153990 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:27.153957 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5ff336-7d447bd878-v7jmb"] Nov 02 23:30:27.215658 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:27.215623 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a8789b-2ff7-4c73-9bee-2d3879cbe2b9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5ff336-7d447bd878-v7jmb\" (UID: \"89a8789b-2ff7-4c73-9bee-2d3879cbe2b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5ff336-7d447bd878-v7jmb" Nov 02 23:30:27.216097 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:27.216074 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtxlp\" (UniqueName: \"kubernetes.io/projected/89a8789b-2ff7-4c73-9bee-2d3879cbe2b9-kube-api-access-xtxlp\") pod \"payload-processing-e2e-ipp-b-5ff336-7d447bd878-v7jmb\" (UID: \"89a8789b-2ff7-4c73-9bee-2d3879cbe2b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5ff336-7d447bd878-v7jmb" Nov 02 23:30:27.317833 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:27.317735 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtxlp\" (UniqueName: \"kubernetes.io/projected/89a8789b-2ff7-4c73-9bee-2d3879cbe2b9-kube-api-access-xtxlp\") pod \"payload-processing-e2e-ipp-b-5ff336-7d447bd878-v7jmb\" (UID: \"89a8789b-2ff7-4c73-9bee-2d3879cbe2b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5ff336-7d447bd878-v7jmb" Nov 02 23:30:27.318035 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:27.317926 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a8789b-2ff7-4c73-9bee-2d3879cbe2b9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5ff336-7d447bd878-v7jmb\" (UID: \"89a8789b-2ff7-4c73-9bee-2d3879cbe2b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5ff336-7d447bd878-v7jmb" Nov 02 23:30:27.318654 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:27.318628 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a8789b-2ff7-4c73-9bee-2d3879cbe2b9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5ff336-7d447bd878-v7jmb\" (UID: \"89a8789b-2ff7-4c73-9bee-2d3879cbe2b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5ff336-7d447bd878-v7jmb" Nov 02 23:30:27.372145 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:27.372113 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtxlp\" (UniqueName: \"kubernetes.io/projected/89a8789b-2ff7-4c73-9bee-2d3879cbe2b9-kube-api-access-xtxlp\") pod \"payload-processing-e2e-ipp-b-5ff336-7d447bd878-v7jmb\" (UID: \"89a8789b-2ff7-4c73-9bee-2d3879cbe2b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5ff336-7d447bd878-v7jmb" Nov 02 23:30:27.418636 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:27.418600 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5ff336-7d447bd878-v7jmb" Nov 02 23:30:27.785949 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:27.785921 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5ff336-7d447bd878-v7jmb"] Nov 02 23:30:27.787079 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:30:27.787052 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89a8789b_2ff7_4c73_9bee_2d3879cbe2b9.slice/crio-ceb345e51032b5f34a95a65189fcde71014021c3cc69f6ec0ae0a93a867419e6 WatchSource:0}: Error finding container ceb345e51032b5f34a95a65189fcde71014021c3cc69f6ec0ae0a93a867419e6: Status 404 returned error can't find the container with id ceb345e51032b5f34a95a65189fcde71014021c3cc69f6ec0ae0a93a867419e6 Nov 02 23:30:27.995845 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:27.995802 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-5fdb8d5c9f-8xdhn" event={"ID":"a6ef156b-92aa-4950-b83e-294e4adc2e77","Type":"ContainerStarted","Data":"fa0b7f5ace5dd7309266db5f2ebd09b16fb0911167827e14bee90bc76bfae09c"} Nov 02 23:30:27.996080 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:27.995867 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-5fdb8d5c9f-8xdhn" event={"ID":"a6ef156b-92aa-4950-b83e-294e4adc2e77","Type":"ContainerStarted","Data":"f03aa0a8e05df12b5cf7ecae8e87dbac7ecaf4332737c9a1ea75b4c8dd49eea6"} Nov 02 23:30:27.996080 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:27.996010 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-5fdb8d5c9f-8xdhn" Nov 02 23:30:27.997476 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:27.997450 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5ff336-7d447bd878-v7jmb" event={"ID":"89a8789b-2ff7-4c73-9bee-2d3879cbe2b9","Type":"ContainerStarted","Data":"eaf11421bb0c333736917c51669b4f46a8c723acfc901c22d7529dd49bb22b7b"} Nov 02 23:30:27.997608 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:27.997497 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5ff336-7d447bd878-v7jmb" event={"ID":"89a8789b-2ff7-4c73-9bee-2d3879cbe2b9","Type":"ContainerStarted","Data":"ceb345e51032b5f34a95a65189fcde71014021c3cc69f6ec0ae0a93a867419e6"} Nov 02 23:30:27.997608 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:27.997520 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5ff336-7d447bd878-v7jmb" Nov 02 23:30:28.105092 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:28.104955 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-5fdb8d5c9f-8xdhn" podStartSLOduration=2.1049403 podStartE2EDuration="2.1049403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:30:28.104235145 +0000 UTC m=+2277.072766929" watchObservedRunningTime="2026-11-02 23:30:28.1049403 +0000 UTC m=+2277.073472084" Nov 02 23:30:28.218399 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:28.218327 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-5ff336-7d447bd878-v7jmb" podStartSLOduration=1.218307092 podStartE2EDuration="1.218307092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:30:28.213068984 +0000 UTC m=+2277.181600767" watchObservedRunningTime="2026-11-02 23:30:28.218307092 +0000 UTC m=+2277.186838877" Nov 02 23:30:39.006042 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:39.006014 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-5fdb8d5c9f-8xdhn" Nov 02 23:30:39.006479 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:39.006087 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5ff336-7d447bd878-v7jmb" Nov 02 23:30:39.884217 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:30:39.884172 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:30:42.877882 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:42.877798 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94"] Nov 02 23:30:42.883045 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:42.883026 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94" Nov 02 23:30:42.917292 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:42.917257 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3d2ee41-aaf6-46fe-8c0a-1fdff3311197-metrics-certs\") pod \"maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94\" (UID: \"b3d2ee41-aaf6-46fe-8c0a-1fdff3311197\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94" Nov 02 23:30:42.917428 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:42.917312 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3d2ee41-aaf6-46fe-8c0a-1fdff3311197-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94\" (UID: \"b3d2ee41-aaf6-46fe-8c0a-1fdff3311197\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94" Nov 02 23:30:42.917488 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:42.917433 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d2ee41-aaf6-46fe-8c0a-1fdff3311197-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94\" (UID: \"b3d2ee41-aaf6-46fe-8c0a-1fdff3311197\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94" Nov 02 23:30:42.917539 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:42.917525 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhcx5\" (UniqueName: \"kubernetes.io/projected/b3d2ee41-aaf6-46fe-8c0a-1fdff3311197-kube-api-access-vhcx5\") pod \"maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94\" (UID: \"b3d2ee41-aaf6-46fe-8c0a-1fdff3311197\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94" Nov 02 23:30:42.935982 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:42.935952 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94"] Nov 02 23:30:42.979693 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:42.979651 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-5a3909-serving-cert\"" Nov 02 23:30:43.019167 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:43.019131 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3d2ee41-aaf6-46fe-8c0a-1fdff3311197-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94\" (UID: \"b3d2ee41-aaf6-46fe-8c0a-1fdff3311197\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94" Nov 02 23:30:43.019359 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:43.019191 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d2ee41-aaf6-46fe-8c0a-1fdff3311197-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94\" (UID: \"b3d2ee41-aaf6-46fe-8c0a-1fdff3311197\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94" Nov 02 23:30:43.019359 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:43.019321 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhcx5\" (UniqueName: \"kubernetes.io/projected/b3d2ee41-aaf6-46fe-8c0a-1fdff3311197-kube-api-access-vhcx5\") pod \"maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94\" (UID: \"b3d2ee41-aaf6-46fe-8c0a-1fdff3311197\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94" Nov 02 23:30:43.019484 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:43.019471 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3d2ee41-aaf6-46fe-8c0a-1fdff3311197-metrics-certs\") pod \"maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94\" (UID: \"b3d2ee41-aaf6-46fe-8c0a-1fdff3311197\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94" Nov 02 23:30:43.021091 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:43.020415 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d2ee41-aaf6-46fe-8c0a-1fdff3311197-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94\" (UID: \"b3d2ee41-aaf6-46fe-8c0a-1fdff3311197\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94" Nov 02 23:30:43.021091 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:30:43.020530 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a3909-serving-cert: secret "maas-api-e2e-ipp-a-5a3909-serving-cert" not found Nov 02 23:30:43.021091 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:30:43.020590 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3d2ee41-aaf6-46fe-8c0a-1fdff3311197-maas-api-tls podName:b3d2ee41-aaf6-46fe-8c0a-1fdff3311197 nodeName:}" failed. No retries permitted until 2026-11-02 23:30:43.520571486 +0000 UTC m=+2292.489103266 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b3d2ee41-aaf6-46fe-8c0a-1fdff3311197-maas-api-tls") pod "maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94" (UID: "b3d2ee41-aaf6-46fe-8c0a-1fdff3311197") : secret "maas-api-e2e-ipp-a-5a3909-serving-cert" not found Nov 02 23:30:43.025820 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:43.025768 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3d2ee41-aaf6-46fe-8c0a-1fdff3311197-metrics-certs\") pod \"maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94\" (UID: \"b3d2ee41-aaf6-46fe-8c0a-1fdff3311197\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94" Nov 02 23:30:43.077952 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:43.077881 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhcx5\" (UniqueName: \"kubernetes.io/projected/b3d2ee41-aaf6-46fe-8c0a-1fdff3311197-kube-api-access-vhcx5\") pod \"maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94\" (UID: \"b3d2ee41-aaf6-46fe-8c0a-1fdff3311197\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94" Nov 02 23:30:43.531558 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:43.531508 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3d2ee41-aaf6-46fe-8c0a-1fdff3311197-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94\" (UID: \"b3d2ee41-aaf6-46fe-8c0a-1fdff3311197\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94" Nov 02 23:30:43.534113 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:43.534078 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3d2ee41-aaf6-46fe-8c0a-1fdff3311197-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94\" (UID: \"b3d2ee41-aaf6-46fe-8c0a-1fdff3311197\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94" Nov 02 23:30:43.797298 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:43.797204 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94" Nov 02 23:30:44.175736 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:44.175700 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94"] Nov 02 23:30:44.176177 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:30:44.175948 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3d2ee41_aaf6_46fe_8c0a_1fdff3311197.slice/crio-6754eaa04ee8ad56723b9e304ed9f0ac3281ec9a89e6ec67e307462b61edcb18 WatchSource:0}: Error finding container 6754eaa04ee8ad56723b9e304ed9f0ac3281ec9a89e6ec67e307462b61edcb18: Status 404 returned error can't find the container with id 6754eaa04ee8ad56723b9e304ed9f0ac3281ec9a89e6ec67e307462b61edcb18 Nov 02 23:30:45.101498 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:45.101454 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94" event={"ID":"b3d2ee41-aaf6-46fe-8c0a-1fdff3311197","Type":"ContainerStarted","Data":"6754eaa04ee8ad56723b9e304ed9f0ac3281ec9a89e6ec67e307462b61edcb18"} Nov 02 23:30:46.109092 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:46.109052 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94" event={"ID":"b3d2ee41-aaf6-46fe-8c0a-1fdff3311197","Type":"ContainerStarted","Data":"e7297e705d87bac61193ce57ce37d9dcdbd515b602754e9da23b7952795ee977"} Nov 02 23:30:46.109494 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:46.109165 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94" Nov 02 23:30:46.220103 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:46.220049 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94" podStartSLOduration=3.261621936 podStartE2EDuration="4.220033691s" totalImagesPullingTime="958.411755ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:30:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:30:46.218977573 +0000 UTC m=+2295.187509358" watchObservedRunningTime="2026-11-02 23:30:46.220033691 +0000 UTC m=+2295.188565475" Nov 02 23:30:52.121124 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:52.121090 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94" Nov 02 23:30:52.724395 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:52.724363 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z"] Nov 02 23:30:52.728747 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:52.728726 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z" Nov 02 23:30:52.779364 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:52.779326 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-fb5fa2-serving-cert\"" Nov 02 23:30:52.783341 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:52.783308 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z"] Nov 02 23:30:52.850047 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:52.850020 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ac3a7a9-9008-4869-8067-c9e82ad9a79f-metrics-certs\") pod \"maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z\" (UID: \"6ac3a7a9-9008-4869-8067-c9e82ad9a79f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z" Nov 02 23:30:52.850243 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:52.850197 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ac3a7a9-9008-4869-8067-c9e82ad9a79f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z\" (UID: \"6ac3a7a9-9008-4869-8067-c9e82ad9a79f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z" Nov 02 23:30:52.850311 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:52.850264 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ac3a7a9-9008-4869-8067-c9e82ad9a79f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z\" (UID: \"6ac3a7a9-9008-4869-8067-c9e82ad9a79f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z" Nov 02 23:30:52.850376 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:52.850357 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc4hl\" (UniqueName: \"kubernetes.io/projected/6ac3a7a9-9008-4869-8067-c9e82ad9a79f-kube-api-access-lc4hl\") pod \"maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z\" (UID: \"6ac3a7a9-9008-4869-8067-c9e82ad9a79f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z" Nov 02 23:30:52.951661 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:52.951621 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ac3a7a9-9008-4869-8067-c9e82ad9a79f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z\" (UID: \"6ac3a7a9-9008-4869-8067-c9e82ad9a79f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z" Nov 02 23:30:52.951938 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:52.951695 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ac3a7a9-9008-4869-8067-c9e82ad9a79f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z\" (UID: \"6ac3a7a9-9008-4869-8067-c9e82ad9a79f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z" Nov 02 23:30:52.951938 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:52.951717 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc4hl\" (UniqueName: \"kubernetes.io/projected/6ac3a7a9-9008-4869-8067-c9e82ad9a79f-kube-api-access-lc4hl\") pod \"maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z\" (UID: \"6ac3a7a9-9008-4869-8067-c9e82ad9a79f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z" Nov 02 23:30:52.951938 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:52.951752 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ac3a7a9-9008-4869-8067-c9e82ad9a79f-metrics-certs\") pod \"maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z\" (UID: \"6ac3a7a9-9008-4869-8067-c9e82ad9a79f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z" Nov 02 23:30:52.951938 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:30:52.951770 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-fb5fa2-serving-cert: secret "maas-api-e2e-praxis-a-fb5fa2-serving-cert" not found Nov 02 23:30:52.951938 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:30:52.951849 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ac3a7a9-9008-4869-8067-c9e82ad9a79f-maas-api-tls podName:6ac3a7a9-9008-4869-8067-c9e82ad9a79f nodeName:}" failed. No retries permitted until 2026-11-02 23:30:53.451827824 +0000 UTC m=+2302.420359605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6ac3a7a9-9008-4869-8067-c9e82ad9a79f-maas-api-tls") pod "maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z" (UID: "6ac3a7a9-9008-4869-8067-c9e82ad9a79f") : secret "maas-api-e2e-praxis-a-fb5fa2-serving-cert" not found Nov 02 23:30:52.952498 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:52.952473 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ac3a7a9-9008-4869-8067-c9e82ad9a79f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z\" (UID: \"6ac3a7a9-9008-4869-8067-c9e82ad9a79f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z" Nov 02 23:30:52.954336 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:52.954312 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ac3a7a9-9008-4869-8067-c9e82ad9a79f-metrics-certs\") pod \"maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z\" (UID: \"6ac3a7a9-9008-4869-8067-c9e82ad9a79f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z" Nov 02 23:30:53.007719 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:53.007651 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc4hl\" (UniqueName: \"kubernetes.io/projected/6ac3a7a9-9008-4869-8067-c9e82ad9a79f-kube-api-access-lc4hl\") pod \"maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z\" (UID: \"6ac3a7a9-9008-4869-8067-c9e82ad9a79f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z" Nov 02 23:30:53.459542 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:53.459488 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ac3a7a9-9008-4869-8067-c9e82ad9a79f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z\" (UID: \"6ac3a7a9-9008-4869-8067-c9e82ad9a79f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z" Nov 02 23:30:53.461955 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:53.461927 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ac3a7a9-9008-4869-8067-c9e82ad9a79f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z\" (UID: \"6ac3a7a9-9008-4869-8067-c9e82ad9a79f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z" Nov 02 23:30:53.642356 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:53.642318 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z" Nov 02 23:30:54.010210 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:54.010170 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z"] Nov 02 23:30:54.011205 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:30:54.011177 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac3a7a9_9008_4869_8067_c9e82ad9a79f.slice/crio-1b063c3e49ed7806e54d9fb4866d1abf6affe32122dfc6607b2d34f2060b2294 WatchSource:0}: Error finding container 1b063c3e49ed7806e54d9fb4866d1abf6affe32122dfc6607b2d34f2060b2294: Status 404 returned error can't find the container with id 1b063c3e49ed7806e54d9fb4866d1abf6affe32122dfc6607b2d34f2060b2294 Nov 02 23:30:54.167261 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:54.167217 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z" event={"ID":"6ac3a7a9-9008-4869-8067-c9e82ad9a79f","Type":"ContainerStarted","Data":"1b063c3e49ed7806e54d9fb4866d1abf6affe32122dfc6607b2d34f2060b2294"} Nov 02 23:30:55.174660 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:55.174579 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z" event={"ID":"6ac3a7a9-9008-4869-8067-c9e82ad9a79f","Type":"ContainerStarted","Data":"1e2a3fb83bebb99b18716f99dcb4b984e0f2ec9f376c7fa9a5f7ffd1c4a8b399"} Nov 02 23:30:55.175076 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:55.174720 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z" Nov 02 23:30:55.284799 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:30:55.284743 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z" podStartSLOduration=2.374843997 podStartE2EDuration="3.284724615s" totalImagesPullingTime="909.880618ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:30:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:30:55.283775677 +0000 UTC m=+2304.252307462" watchObservedRunningTime="2026-11-02 23:30:55.284724615 +0000 UTC m=+2304.253256402" Nov 02 23:31:01.187181 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:31:01.187151 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z" Nov 02 23:31:09.885095 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:31:09.885058 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:31:21.388971 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:31:21.388938 5076 generic.go:356] "Generic (PLEG): container finished" podID="e8d06dd9-47b5-442a-ad96-f8616e6570a8" containerID="f01d415c678dd1f861af35a82f6eb8c5b5bb378a8770c175d19b4360b25ff7d5" exitCode=0 Nov 02 23:31:21.389443 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:31:21.389014 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" event={"ID":"e8d06dd9-47b5-442a-ad96-f8616e6570a8","Type":"ContainerDied","Data":"f01d415c678dd1f861af35a82f6eb8c5b5bb378a8770c175d19b4360b25ff7d5"} Nov 02 23:31:21.389498 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:31:21.389464 5076 scope.go:122] "RemoveContainer" containerID="f01d415c678dd1f861af35a82f6eb8c5b5bb378a8770c175d19b4360b25ff7d5" Nov 02 23:31:21.391244 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:31:21.391212 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:31:21.391344 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:31:21.391283 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:31:21.391344 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:31:21.391316 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:31:22.396975 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:31:22.396934 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" event={"ID":"e8d06dd9-47b5-442a-ad96-f8616e6570a8","Type":"ContainerStarted","Data":"f470c2f29506f4d1d48af594511abf37620bccdc141ad07f1f43255ff8aa1aec"} Nov 02 23:31:23.003570 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:31:23.003525 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" Nov 02 23:31:23.005098 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:31:23.005071 5076 patch_prober.go:29] interesting pod/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 02 23:31:23.005222 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:31:23.005121 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" podUID="e8d06dd9-47b5-442a-ad96-f8616e6570a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 02 23:31:24.002898 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:31:24.002855 5076 patch_prober.go:29] interesting pod/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 02 23:31:24.003289 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:31:24.002926 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" podUID="e8d06dd9-47b5-442a-ad96-f8616e6570a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 02 23:31:25.003706 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:31:25.003666 5076 patch_prober.go:29] interesting pod/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 02 23:31:25.004158 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:31:25.003723 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" podUID="e8d06dd9-47b5-442a-ad96-f8616e6570a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 02 23:31:25.580467 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:31:25.580431 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" Nov 02 23:31:26.003513 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:31:26.003479 5076 patch_prober.go:29] interesting pod/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 02 23:31:26.003706 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:31:26.003541 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" podUID="e8d06dd9-47b5-442a-ad96-f8616e6570a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 02 23:31:27.003136 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:31:27.003101 5076 patch_prober.go:29] interesting pod/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 02 23:31:27.003541 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:31:27.003205 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" podUID="e8d06dd9-47b5-442a-ad96-f8616e6570a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 02 23:31:28.008445 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:31:28.008413 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" Nov 02 23:31:28.009737 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:31:28.009719 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" Nov 02 23:31:39.885951 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:31:39.885881 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:32:08.551206 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:08.551168 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp"] Nov 02 23:32:08.551752 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:08.551393 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" podUID="6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91" containerName="istio-proxy" containerID="cri-o://8a88dbbc1feb4213a82a24bd49011011a93468b93bc25bf7580d117753d65525" gracePeriod=30 Nov 02 23:32:09.886019 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:32:09.885983 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:32:13.604110 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:13.604078 5076 patch_prober.go:29] interesting pod/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 23:32:13.604472 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:13.604145 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" podUID="6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 23:32:13.744252 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:13.744220 5076 generic.go:356] "Generic (PLEG): container finished" podID="6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91" containerID="8a88dbbc1feb4213a82a24bd49011011a93468b93bc25bf7580d117753d65525" exitCode=0 Nov 02 23:32:13.744404 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:13.744357 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" event={"ID":"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91","Type":"ContainerDied","Data":"8a88dbbc1feb4213a82a24bd49011011a93468b93bc25bf7580d117753d65525"} Nov 02 23:32:13.744449 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:13.744405 5076 scope.go:122] "RemoveContainer" containerID="e5d4bf19db2a1293fb208a5ebda6a1e629e4289d83626d9b996a3e7fe2fbb601" Nov 02 23:32:13.837468 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:13.837403 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" Nov 02 23:32:13.999640 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:13.999600 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-workload-certs\") pod \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\" (UID: \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\") " Nov 02 23:32:13.999836 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:13.999719 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-istio-token\" (UniqueName: \"kubernetes.io/projected/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-istio-token\") pod \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\" (UID: \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\") " Nov 02 23:32:13.999836 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:13.999757 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-istio-podinfo\") pod \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\" (UID: \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\") " Nov 02 23:32:13.999836 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:13.999785 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-istiod-ca-cert\") pod \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\" (UID: \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\") " Nov 02 23:32:14.000037 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:13.999840 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-credential-socket\") pod \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\" (UID: \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\") " Nov 02 23:32:14.000037 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:13.999884 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-workload-socket\") pod \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\" (UID: \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\") " Nov 02 23:32:14.000037 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:14.000006 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-istio-data\") pod \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\" (UID: \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\") " Nov 02 23:32:14.000208 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:14.000075 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-istio-envoy\") pod \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\" (UID: \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\") " Nov 02 23:32:14.000208 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:14.000127 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-kube-api-access-sp9m4\" (UniqueName: \"kubernetes.io/projected/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-kube-api-access-sp9m4\") pod \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\" (UID: \"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91\") " Nov 02 23:32:14.001138 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:14.001109 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-workload-socket" pod "6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91" (UID: "6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:32:14.001324 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:14.001210 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-istio-data" pod "6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91" (UID: "6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:32:14.001324 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:14.001229 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-istiod-ca-cert" pod "6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91" (UID: "6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:32:14.001518 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:14.001424 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-workload-certs" pod "6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91" (UID: "6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:32:14.001518 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:14.001496 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-credential-socket" pod "6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91" (UID: "6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:32:14.003326 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:14.003300 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-kube-api-access-sp9m4" pod "6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91" (UID: "6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91"). InnerVolumeSpecName "kube-api-access-sp9m4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:32:14.003508 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:14.003486 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-istio-podinfo" pod "6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91" (UID: "6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:32:14.003706 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:14.003686 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-istio-token" pod "6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91" (UID: "6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:32:14.003807 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:14.003788 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-istio-envoy" pod "6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91" (UID: "6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:32:14.101574 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:14.101487 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:32:14.101574 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:14.101517 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:32:14.101574 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:14.101526 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:32:14.101574 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:14.101536 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp9m4\" (UniqueName: \"kubernetes.io/projected/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-kube-api-access-sp9m4\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:32:14.101574 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:14.101545 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:32:14.101574 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:14.101553 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:32:14.101574 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:14.101562 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:32:14.101574 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:14.101571 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:32:14.101574 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:14.101579 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:32:14.761764 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:14.761716 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" event={"ID":"6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91","Type":"ContainerDied","Data":"51a50052ee7c65a7a187510ad51446d165d4a95d84cc2ac017cbf697bd5e2cc4"} Nov 02 23:32:14.762355 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:14.761800 5076 scope.go:122] "RemoveContainer" containerID="8a88dbbc1feb4213a82a24bd49011011a93468b93bc25bf7580d117753d65525" Nov 02 23:32:14.762355 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:14.761972 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp" Nov 02 23:32:14.923937 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:14.923866 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp"] Nov 02 23:32:14.971261 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:14.971227 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-3bf62f-openshift-default-745b84b9c-rcmjp"] Nov 02 23:32:15.179989 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:15.179323 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91" path="/var/lib/kubelet/pods/6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91/volumes" Nov 02 23:32:20.883611 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:20.883572 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9"] Nov 02 23:32:20.884095 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:20.883817 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9" podUID="0e8ca80b-ef9c-4321-a986-b832858f7fd4" containerName="maas-api" containerID="cri-o://544680aacea4ae58955c718ed54b6184ea3ca2bd4f62d6c4cc0969c6234dcad7" gracePeriod=30 Nov 02 23:32:21.188822 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:21.188794 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9" Nov 02 23:32:21.209099 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:21.209071 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e8ca80b-ef9c-4321-a986-b832858f7fd4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e8ca80b-ef9c-4321-a986-b832858f7fd4-metrics-certs\") pod \"0e8ca80b-ef9c-4321-a986-b832858f7fd4\" (UID: \"0e8ca80b-ef9c-4321-a986-b832858f7fd4\") " Nov 02 23:32:21.209231 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:21.209165 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e8ca80b-ef9c-4321-a986-b832858f7fd4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e8ca80b-ef9c-4321-a986-b832858f7fd4-maas-api-tls\") pod \"0e8ca80b-ef9c-4321-a986-b832858f7fd4\" (UID: \"0e8ca80b-ef9c-4321-a986-b832858f7fd4\") " Nov 02 23:32:21.209231 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:21.209226 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e8ca80b-ef9c-4321-a986-b832858f7fd4-kube-api-access-dh8r6\" (UniqueName: \"kubernetes.io/projected/0e8ca80b-ef9c-4321-a986-b832858f7fd4-kube-api-access-dh8r6\") pod \"0e8ca80b-ef9c-4321-a986-b832858f7fd4\" (UID: \"0e8ca80b-ef9c-4321-a986-b832858f7fd4\") " Nov 02 23:32:21.209300 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:21.209274 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e8ca80b-ef9c-4321-a986-b832858f7fd4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e8ca80b-ef9c-4321-a986-b832858f7fd4-openshift-service-ca\") pod \"0e8ca80b-ef9c-4321-a986-b832858f7fd4\" (UID: \"0e8ca80b-ef9c-4321-a986-b832858f7fd4\") " Nov 02 23:32:21.209676 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:21.209648 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e8ca80b-ef9c-4321-a986-b832858f7fd4-openshift-service-ca" pod "0e8ca80b-ef9c-4321-a986-b832858f7fd4" (UID: "0e8ca80b-ef9c-4321-a986-b832858f7fd4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:32:21.211221 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:21.211199 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e8ca80b-ef9c-4321-a986-b832858f7fd4-kube-api-access-dh8r6" pod "0e8ca80b-ef9c-4321-a986-b832858f7fd4" (UID: "0e8ca80b-ef9c-4321-a986-b832858f7fd4"). InnerVolumeSpecName "kube-api-access-dh8r6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:32:21.211580 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:21.211562 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e8ca80b-ef9c-4321-a986-b832858f7fd4-maas-api-tls" pod "0e8ca80b-ef9c-4321-a986-b832858f7fd4" (UID: "0e8ca80b-ef9c-4321-a986-b832858f7fd4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:32:21.211649 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:21.211577 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e8ca80b-ef9c-4321-a986-b832858f7fd4-metrics-certs" pod "0e8ca80b-ef9c-4321-a986-b832858f7fd4" (UID: "0e8ca80b-ef9c-4321-a986-b832858f7fd4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:32:21.310539 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:21.310495 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e8ca80b-ef9c-4321-a986-b832858f7fd4-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:32:21.310539 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:21.310535 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh8r6\" (UniqueName: \"kubernetes.io/projected/0e8ca80b-ef9c-4321-a986-b832858f7fd4-kube-api-access-dh8r6\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:32:21.310793 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:21.310551 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e8ca80b-ef9c-4321-a986-b832858f7fd4-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:32:21.310793 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:21.310567 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e8ca80b-ef9c-4321-a986-b832858f7fd4-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:32:21.805117 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:21.805077 5076 generic.go:356] "Generic (PLEG): container finished" podID="0e8ca80b-ef9c-4321-a986-b832858f7fd4" containerID="544680aacea4ae58955c718ed54b6184ea3ca2bd4f62d6c4cc0969c6234dcad7" exitCode=0 Nov 02 23:32:21.805299 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:21.805128 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9" Nov 02 23:32:21.805299 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:21.805162 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9" event={"ID":"0e8ca80b-ef9c-4321-a986-b832858f7fd4","Type":"ContainerDied","Data":"544680aacea4ae58955c718ed54b6184ea3ca2bd4f62d6c4cc0969c6234dcad7"} Nov 02 23:32:21.805299 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:21.805222 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9" event={"ID":"0e8ca80b-ef9c-4321-a986-b832858f7fd4","Type":"ContainerDied","Data":"36fab331de035d9aa0dbe8e85bec0511f129f19ba2c09bdd6fe1b04751cc1202"} Nov 02 23:32:21.805299 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:21.805245 5076 scope.go:122] "RemoveContainer" containerID="544680aacea4ae58955c718ed54b6184ea3ca2bd4f62d6c4cc0969c6234dcad7" Nov 02 23:32:21.816998 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:21.816978 5076 scope.go:122] "RemoveContainer" containerID="544680aacea4ae58955c718ed54b6184ea3ca2bd4f62d6c4cc0969c6234dcad7" Nov 02 23:32:21.817277 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:32:21.817257 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"544680aacea4ae58955c718ed54b6184ea3ca2bd4f62d6c4cc0969c6234dcad7\": container with ID starting with 544680aacea4ae58955c718ed54b6184ea3ca2bd4f62d6c4cc0969c6234dcad7 not found: ID does not exist" containerID="544680aacea4ae58955c718ed54b6184ea3ca2bd4f62d6c4cc0969c6234dcad7" Nov 02 23:32:21.817329 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:21.817288 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"544680aacea4ae58955c718ed54b6184ea3ca2bd4f62d6c4cc0969c6234dcad7"} err="failed to get container status \"544680aacea4ae58955c718ed54b6184ea3ca2bd4f62d6c4cc0969c6234dcad7\": rpc error: code = NotFound desc = could not find container \"544680aacea4ae58955c718ed54b6184ea3ca2bd4f62d6c4cc0969c6234dcad7\": container with ID starting with 544680aacea4ae58955c718ed54b6184ea3ca2bd4f62d6c4cc0969c6234dcad7 not found: ID does not exist" Nov 02 23:32:21.972354 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:21.972316 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9"] Nov 02 23:32:22.004828 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:22.004787 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-02cdd9-5874644f54-wq5z9"] Nov 02 23:32:23.151458 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:23.151416 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e8ca80b-ef9c-4321-a986-b832858f7fd4" path="/var/lib/kubelet/pods/0e8ca80b-ef9c-4321-a986-b832858f7fd4/volumes" Nov 02 23:32:31.795182 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:31.795152 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 02 23:32:31.838709 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:32:31.838683 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 02 23:32:39.885141 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:32:39.885101 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:33:09.735676 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:09.735636 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-7f5cc8bc5-q4q99"] Nov 02 23:33:09.736224 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:09.736208 5076 cpu_manager.go:414] "containerMap: removing container" podUID="6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91" containerName="istio-proxy" Nov 02 23:33:09.736269 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:09.736225 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91" containerName="istio-proxy" Nov 02 23:33:09.736269 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:09.736244 5076 cpu_manager.go:414] "containerMap: removing container" podUID="6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91" containerName="istio-proxy" Nov 02 23:33:09.736269 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:09.736249 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91" containerName="istio-proxy" Nov 02 23:33:09.736366 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:09.736279 5076 cpu_manager.go:414] "containerMap: removing container" podUID="0e8ca80b-ef9c-4321-a986-b832858f7fd4" containerName="maas-api" Nov 02 23:33:09.736366 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:09.736284 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e8ca80b-ef9c-4321-a986-b832858f7fd4" containerName="maas-api" Nov 02 23:33:09.736366 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:09.736359 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e8ca80b-ef9c-4321-a986-b832858f7fd4" containerName="maas-api" Nov 02 23:33:09.736457 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:09.736371 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91" containerName="istio-proxy" Nov 02 23:33:09.736457 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:09.736377 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fa02b25-4e1a-4c98-bfaa-8fa3b0372f91" containerName="istio-proxy" Nov 02 23:33:09.739957 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:09.739934 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-7f5cc8bc5-q4q99" Nov 02 23:33:09.793521 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:09.793477 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-7f5cc8bc5-q4q99"] Nov 02 23:33:09.836963 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:09.836500 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5a3909\"" Nov 02 23:33:09.836963 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:09.836655 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5a3909-dockercfg-6zc4h\"" Nov 02 23:33:09.884037 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:33:09.883995 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:33:09.932039 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:09.932000 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk7dh\" (UniqueName: \"kubernetes.io/projected/41f08c9c-6fe4-4a4d-a633-9dce1493b831-kube-api-access-wk7dh\") pod \"payload-pre-processing-e2e-ipp-a-5a3909-7f5cc8bc5-q4q99\" (UID: \"41f08c9c-6fe4-4a4d-a633-9dce1493b831\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-7f5cc8bc5-q4q99" Nov 02 23:33:09.932217 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:09.932067 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41f08c9c-6fe4-4a4d-a633-9dce1493b831-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5a3909-7f5cc8bc5-q4q99\" (UID: \"41f08c9c-6fe4-4a4d-a633-9dce1493b831\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-7f5cc8bc5-q4q99" Nov 02 23:33:10.033804 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:10.033717 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk7dh\" (UniqueName: \"kubernetes.io/projected/41f08c9c-6fe4-4a4d-a633-9dce1493b831-kube-api-access-wk7dh\") pod \"payload-pre-processing-e2e-ipp-a-5a3909-7f5cc8bc5-q4q99\" (UID: \"41f08c9c-6fe4-4a4d-a633-9dce1493b831\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-7f5cc8bc5-q4q99" Nov 02 23:33:10.033804 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:10.033780 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41f08c9c-6fe4-4a4d-a633-9dce1493b831-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5a3909-7f5cc8bc5-q4q99\" (UID: \"41f08c9c-6fe4-4a4d-a633-9dce1493b831\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-7f5cc8bc5-q4q99" Nov 02 23:33:10.034529 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:10.034509 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41f08c9c-6fe4-4a4d-a633-9dce1493b831-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5a3909-7f5cc8bc5-q4q99\" (UID: \"41f08c9c-6fe4-4a4d-a633-9dce1493b831\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-7f5cc8bc5-q4q99" Nov 02 23:33:10.088154 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:10.088114 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk7dh\" (UniqueName: \"kubernetes.io/projected/41f08c9c-6fe4-4a4d-a633-9dce1493b831-kube-api-access-wk7dh\") pod \"payload-pre-processing-e2e-ipp-a-5a3909-7f5cc8bc5-q4q99\" (UID: \"41f08c9c-6fe4-4a4d-a633-9dce1493b831\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-7f5cc8bc5-q4q99" Nov 02 23:33:10.352851 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:10.352776 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-7f5cc8bc5-q4q99" Nov 02 23:33:10.721490 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:10.721460 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-7f5cc8bc5-q4q99"] Nov 02 23:33:10.722672 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:33:10.722633 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f08c9c_6fe4_4a4d_a633_9dce1493b831.slice/crio-813faf094b3c546ade77812b631b4c3d39cca77d6f284e87c03c7acf4c878db5 WatchSource:0}: Error finding container 813faf094b3c546ade77812b631b4c3d39cca77d6f284e87c03c7acf4c878db5: Status 404 returned error can't find the container with id 813faf094b3c546ade77812b631b4c3d39cca77d6f284e87c03c7acf4c878db5 Nov 02 23:33:11.149649 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:11.149566 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-7f5cc8bc5-q4q99" event={"ID":"41f08c9c-6fe4-4a4d-a633-9dce1493b831","Type":"ContainerStarted","Data":"810ee7a4f8c98c00bcfe6ed479ac1b7972563823d298615a58eceab32ef24f8d"} Nov 02 23:33:11.149649 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:11.149620 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-7f5cc8bc5-q4q99" event={"ID":"41f08c9c-6fe4-4a4d-a633-9dce1493b831","Type":"ContainerStarted","Data":"813faf094b3c546ade77812b631b4c3d39cca77d6f284e87c03c7acf4c878db5"} Nov 02 23:33:11.149649 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:11.149639 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-7f5cc8bc5-q4q99" Nov 02 23:33:11.256545 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:11.256491 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-7f5cc8bc5-q4q99" podStartSLOduration=2.256478877 podStartE2EDuration="2.256478877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:33:11.255640211 +0000 UTC m=+2440.224172009" watchObservedRunningTime="2026-11-02 23:33:11.256478877 +0000 UTC m=+2440.225010678" Nov 02 23:33:20.805665 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:20.805629 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76"] Nov 02 23:33:20.809766 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:20.809744 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76" Nov 02 23:33:20.852320 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:20.852285 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76\" (UID: \"3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76" Nov 02 23:33:20.852527 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:20.852506 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766-maas-api-tls\") pod \"maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76\" (UID: \"3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76" Nov 02 23:33:20.852597 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:20.852558 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766-metrics-certs\") pod \"maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76\" (UID: \"3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76" Nov 02 23:33:20.852680 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:20.852659 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2f4p\" (UniqueName: \"kubernetes.io/projected/3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766-kube-api-access-b2f4p\") pod \"maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76\" (UID: \"3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76" Nov 02 23:33:20.859410 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:20.859371 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-db5d87-serving-cert\"" Nov 02 23:33:20.909974 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:20.909940 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76"] Nov 02 23:33:20.953445 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:20.953411 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766-metrics-certs\") pod \"maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76\" (UID: \"3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76" Nov 02 23:33:20.953637 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:20.953466 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2f4p\" (UniqueName: \"kubernetes.io/projected/3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766-kube-api-access-b2f4p\") pod \"maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76\" (UID: \"3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76" Nov 02 23:33:20.953637 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:20.953516 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76\" (UID: \"3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76" Nov 02 23:33:20.953767 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:20.953665 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766-maas-api-tls\") pod \"maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76\" (UID: \"3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76" Nov 02 23:33:20.953826 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:33:20.953772 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5d87-serving-cert: secret "maas-api-e2e-ipp-b-db5d87-serving-cert" not found Nov 02 23:33:20.953884 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:33:20.953830 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766-maas-api-tls podName:3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766 nodeName:}" failed. No retries permitted until 2026-11-02 23:33:21.453812357 +0000 UTC m=+2450.422344119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766-maas-api-tls") pod "maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76" (UID: "3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766") : secret "maas-api-e2e-ipp-b-db5d87-serving-cert" not found Nov 02 23:33:20.954385 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:20.954352 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76\" (UID: \"3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76" Nov 02 23:33:20.955999 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:20.955968 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766-metrics-certs\") pod \"maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76\" (UID: \"3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76" Nov 02 23:33:21.009001 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:21.008970 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2f4p\" (UniqueName: \"kubernetes.io/projected/3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766-kube-api-access-b2f4p\") pod \"maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76\" (UID: \"3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76" Nov 02 23:33:21.472832 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:21.472798 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766-maas-api-tls\") pod \"maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76\" (UID: \"3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76" Nov 02 23:33:21.475240 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:21.475213 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766-maas-api-tls\") pod \"maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76\" (UID: \"3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76" Nov 02 23:33:21.723040 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:21.722962 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76" Nov 02 23:33:22.090929 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:22.090867 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76"] Nov 02 23:33:22.154306 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:22.154273 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-7f5cc8bc5-q4q99" Nov 02 23:33:22.239546 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:22.239499 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76" event={"ID":"3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766","Type":"ContainerStarted","Data":"1fb6278f27046a7d3ead0e295899512fabc8c3f7c6b010c657fd8750d3acef0e"} Nov 02 23:33:24.252512 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:24.252475 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76" event={"ID":"3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766","Type":"ContainerStarted","Data":"a9c5024d89b7843c06e82166e90ce77716026d40f1701386e8d05eb1340075df"} Nov 02 23:33:24.252905 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:24.252607 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76" Nov 02 23:33:24.365283 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:24.365232 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76" podStartSLOduration=3.252320639 podStartE2EDuration="4.365219008s" totalImagesPullingTime="1.112898369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:33:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:33:24.363442259 +0000 UTC m=+2453.331974042" watchObservedRunningTime="2026-11-02 23:33:24.365219008 +0000 UTC m=+2453.333750792" Nov 02 23:33:30.264734 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:30.264704 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76" Nov 02 23:33:39.884008 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:33:39.883967 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:33:49.483642 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:49.483557 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9"] Nov 02 23:33:49.488827 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:49.488804 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" Nov 02 23:33:49.538745 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:49.538715 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-ba1918-openshift-default-dockercfg-k6bc8\"" Nov 02 23:33:49.591163 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:49.591128 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9"] Nov 02 23:33:49.633464 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:49.633426 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61e6dcd-7996-4565-8481-aaafb4194707-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9\" (UID: \"d61e6dcd-7996-4565-8481-aaafb4194707\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" Nov 02 23:33:49.633632 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:49.633552 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61e6dcd-7996-4565-8481-aaafb4194707-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9\" (UID: \"d61e6dcd-7996-4565-8481-aaafb4194707\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" Nov 02 23:33:49.633632 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:49.633593 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61e6dcd-7996-4565-8481-aaafb4194707-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9\" (UID: \"d61e6dcd-7996-4565-8481-aaafb4194707\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" Nov 02 23:33:49.633717 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:49.633683 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61e6dcd-7996-4565-8481-aaafb4194707-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9\" (UID: \"d61e6dcd-7996-4565-8481-aaafb4194707\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" Nov 02 23:33:49.633752 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:49.633743 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wkpc\" (UniqueName: \"kubernetes.io/projected/d61e6dcd-7996-4565-8481-aaafb4194707-kube-api-access-4wkpc\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9\" (UID: \"d61e6dcd-7996-4565-8481-aaafb4194707\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" Nov 02 23:33:49.633785 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:49.633770 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61e6dcd-7996-4565-8481-aaafb4194707-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9\" (UID: \"d61e6dcd-7996-4565-8481-aaafb4194707\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" Nov 02 23:33:49.633819 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:49.633799 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61e6dcd-7996-4565-8481-aaafb4194707-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9\" (UID: \"d61e6dcd-7996-4565-8481-aaafb4194707\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" Nov 02 23:33:49.633878 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:49.633867 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d61e6dcd-7996-4565-8481-aaafb4194707-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9\" (UID: \"d61e6dcd-7996-4565-8481-aaafb4194707\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" Nov 02 23:33:49.633939 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:49.633928 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61e6dcd-7996-4565-8481-aaafb4194707-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9\" (UID: \"d61e6dcd-7996-4565-8481-aaafb4194707\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" Nov 02 23:33:49.734913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:49.734808 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d61e6dcd-7996-4565-8481-aaafb4194707-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9\" (UID: \"d61e6dcd-7996-4565-8481-aaafb4194707\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" Nov 02 23:33:49.734913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:49.734866 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61e6dcd-7996-4565-8481-aaafb4194707-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9\" (UID: \"d61e6dcd-7996-4565-8481-aaafb4194707\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" Nov 02 23:33:49.735129 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:49.734948 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61e6dcd-7996-4565-8481-aaafb4194707-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9\" (UID: \"d61e6dcd-7996-4565-8481-aaafb4194707\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" Nov 02 23:33:49.735129 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:49.734996 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61e6dcd-7996-4565-8481-aaafb4194707-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9\" (UID: \"d61e6dcd-7996-4565-8481-aaafb4194707\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" Nov 02 23:33:49.735129 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:49.735015 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61e6dcd-7996-4565-8481-aaafb4194707-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9\" (UID: \"d61e6dcd-7996-4565-8481-aaafb4194707\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" Nov 02 23:33:49.735129 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:49.735092 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61e6dcd-7996-4565-8481-aaafb4194707-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9\" (UID: \"d61e6dcd-7996-4565-8481-aaafb4194707\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" Nov 02 23:33:49.735347 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:49.735172 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wkpc\" (UniqueName: \"kubernetes.io/projected/d61e6dcd-7996-4565-8481-aaafb4194707-kube-api-access-4wkpc\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9\" (UID: \"d61e6dcd-7996-4565-8481-aaafb4194707\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" Nov 02 23:33:49.735347 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:49.735220 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61e6dcd-7996-4565-8481-aaafb4194707-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9\" (UID: \"d61e6dcd-7996-4565-8481-aaafb4194707\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" Nov 02 23:33:49.735347 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:49.735270 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61e6dcd-7996-4565-8481-aaafb4194707-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9\" (UID: \"d61e6dcd-7996-4565-8481-aaafb4194707\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" Nov 02 23:33:49.735347 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:49.735319 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61e6dcd-7996-4565-8481-aaafb4194707-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9\" (UID: \"d61e6dcd-7996-4565-8481-aaafb4194707\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" Nov 02 23:33:49.735566 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:49.735377 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61e6dcd-7996-4565-8481-aaafb4194707-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9\" (UID: \"d61e6dcd-7996-4565-8481-aaafb4194707\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" Nov 02 23:33:49.735566 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:49.735491 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61e6dcd-7996-4565-8481-aaafb4194707-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9\" (UID: \"d61e6dcd-7996-4565-8481-aaafb4194707\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" Nov 02 23:33:49.735566 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:49.735509 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61e6dcd-7996-4565-8481-aaafb4194707-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9\" (UID: \"d61e6dcd-7996-4565-8481-aaafb4194707\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" Nov 02 23:33:49.735858 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:49.735838 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61e6dcd-7996-4565-8481-aaafb4194707-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9\" (UID: \"d61e6dcd-7996-4565-8481-aaafb4194707\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" Nov 02 23:33:49.737285 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:49.737253 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61e6dcd-7996-4565-8481-aaafb4194707-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9\" (UID: \"d61e6dcd-7996-4565-8481-aaafb4194707\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" Nov 02 23:33:49.737591 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:49.737575 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61e6dcd-7996-4565-8481-aaafb4194707-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9\" (UID: \"d61e6dcd-7996-4565-8481-aaafb4194707\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" Nov 02 23:33:49.789457 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:49.789425 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d61e6dcd-7996-4565-8481-aaafb4194707-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9\" (UID: \"d61e6dcd-7996-4565-8481-aaafb4194707\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" Nov 02 23:33:49.789605 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:49.789496 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wkpc\" (UniqueName: \"kubernetes.io/projected/d61e6dcd-7996-4565-8481-aaafb4194707-kube-api-access-4wkpc\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9\" (UID: \"d61e6dcd-7996-4565-8481-aaafb4194707\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" Nov 02 23:33:49.803121 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:49.803090 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" Nov 02 23:33:50.221515 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:50.221481 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9"] Nov 02 23:33:50.222840 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:33:50.222808 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd61e6dcd_7996_4565_8481_aaafb4194707.slice/crio-5a8ef275495ff145d7e24678279b261ce5ebce9f7cd45e7836bc3cfca75ed80a WatchSource:0}: Error finding container 5a8ef275495ff145d7e24678279b261ce5ebce9f7cd45e7836bc3cfca75ed80a: Status 404 returned error can't find the container with id 5a8ef275495ff145d7e24678279b261ce5ebce9f7cd45e7836bc3cfca75ed80a Nov 02 23:33:50.225225 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:50.225194 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:33:50.225303 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:50.225258 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:33:50.225303 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:50.225287 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:33:50.414246 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:50.414211 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" event={"ID":"d61e6dcd-7996-4565-8481-aaafb4194707","Type":"ContainerStarted","Data":"e29511754dd30fa3434a74888a9ee05eacc79b4569c7da9629564a7e68c6e281"} Nov 02 23:33:50.414465 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:50.414272 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" event={"ID":"d61e6dcd-7996-4565-8481-aaafb4194707","Type":"ContainerStarted","Data":"5a8ef275495ff145d7e24678279b261ce5ebce9f7cd45e7836bc3cfca75ed80a"} Nov 02 23:33:50.526704 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:50.526590 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" podStartSLOduration=1.526576342 podStartE2EDuration="1.526576342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:33:50.526394641 +0000 UTC m=+2479.494926437" watchObservedRunningTime="2026-11-02 23:33:50.526576342 +0000 UTC m=+2479.495108126" Nov 02 23:33:50.804045 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:50.803961 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" Nov 02 23:33:51.804823 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:51.804785 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 02 23:33:51.805251 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:51.804858 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" podUID="d61e6dcd-7996-4565-8481-aaafb4194707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 02 23:33:52.803619 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:52.803591 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 02 23:33:52.803790 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:52.803654 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" podUID="d61e6dcd-7996-4565-8481-aaafb4194707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 02 23:33:53.804318 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:53.804285 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 02 23:33:53.804717 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:53.804334 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" podUID="d61e6dcd-7996-4565-8481-aaafb4194707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 02 23:33:54.803926 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:54.803877 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 02 23:33:54.804131 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:54.803952 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" podUID="d61e6dcd-7996-4565-8481-aaafb4194707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 02 23:33:55.804410 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:55.804370 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 02 23:33:55.804796 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:55.804428 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" podUID="d61e6dcd-7996-4565-8481-aaafb4194707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 02 23:33:56.803574 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:56.803535 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 02 23:33:56.803746 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:56.803593 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" podUID="d61e6dcd-7996-4565-8481-aaafb4194707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 02 23:33:57.804445 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:57.804403 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 02 23:33:57.804963 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:57.804462 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" podUID="d61e6dcd-7996-4565-8481-aaafb4194707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 02 23:33:59.222922 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:59.222851 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" Nov 02 23:33:59.223422 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:59.223262 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" Nov 02 23:33:59.224158 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:33:59.224135 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" Nov 02 23:34:04.972961 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:34:04.972928 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-981724-d5985fb4-s88v5_d5a2fe82-afb7-4822-b714-d44da311b6ed/payload-processing/0.log" Nov 02 23:34:05.652072 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:34:05.652040 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-5ff336-7d447bd878-v7jmb_89a8789b-2ff7-4c73-9bee-2d3879cbe2b9/payload-processing/0.log" Nov 02 23:34:05.864326 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:34:05.864301 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-5ff336-7d447bd878-v7jmb_89a8789b-2ff7-4c73-9bee-2d3879cbe2b9/payload-processing/0.log" Nov 02 23:34:09.884967 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:34:09.884931 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:34:39.884602 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:34:39.884562 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:35:09.886211 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:35:09.886176 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:35:30.960417 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:30.960371 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94"] Nov 02 23:35:30.960928 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:30.960618 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94" podUID="b3d2ee41-aaf6-46fe-8c0a-1fdff3311197" containerName="maas-api" containerID="cri-o://e7297e705d87bac61193ce57ce37d9dcdbd515b602754e9da23b7952795ee977" gracePeriod=30 Nov 02 23:35:31.117204 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:31.117167 5076 generic.go:356] "Generic (PLEG): container finished" podID="b3d2ee41-aaf6-46fe-8c0a-1fdff3311197" containerID="e7297e705d87bac61193ce57ce37d9dcdbd515b602754e9da23b7952795ee977" exitCode=0 Nov 02 23:35:31.117364 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:31.117198 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94" event={"ID":"b3d2ee41-aaf6-46fe-8c0a-1fdff3311197","Type":"ContainerDied","Data":"e7297e705d87bac61193ce57ce37d9dcdbd515b602754e9da23b7952795ee977"} Nov 02 23:35:31.263793 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:31.263764 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94" Nov 02 23:35:31.299014 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:31.298986 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3d2ee41-aaf6-46fe-8c0a-1fdff3311197-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d2ee41-aaf6-46fe-8c0a-1fdff3311197-openshift-service-ca\") pod \"b3d2ee41-aaf6-46fe-8c0a-1fdff3311197\" (UID: \"b3d2ee41-aaf6-46fe-8c0a-1fdff3311197\") " Nov 02 23:35:31.299192 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:31.299039 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3d2ee41-aaf6-46fe-8c0a-1fdff3311197-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3d2ee41-aaf6-46fe-8c0a-1fdff3311197-maas-api-tls\") pod \"b3d2ee41-aaf6-46fe-8c0a-1fdff3311197\" (UID: \"b3d2ee41-aaf6-46fe-8c0a-1fdff3311197\") " Nov 02 23:35:31.299192 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:31.299084 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3d2ee41-aaf6-46fe-8c0a-1fdff3311197-kube-api-access-vhcx5\" (UniqueName: \"kubernetes.io/projected/b3d2ee41-aaf6-46fe-8c0a-1fdff3311197-kube-api-access-vhcx5\") pod \"b3d2ee41-aaf6-46fe-8c0a-1fdff3311197\" (UID: \"b3d2ee41-aaf6-46fe-8c0a-1fdff3311197\") " Nov 02 23:35:31.299192 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:31.299144 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3d2ee41-aaf6-46fe-8c0a-1fdff3311197-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3d2ee41-aaf6-46fe-8c0a-1fdff3311197-metrics-certs\") pod \"b3d2ee41-aaf6-46fe-8c0a-1fdff3311197\" (UID: \"b3d2ee41-aaf6-46fe-8c0a-1fdff3311197\") " Nov 02 23:35:31.299402 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:31.299373 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3d2ee41-aaf6-46fe-8c0a-1fdff3311197-openshift-service-ca" pod "b3d2ee41-aaf6-46fe-8c0a-1fdff3311197" (UID: "b3d2ee41-aaf6-46fe-8c0a-1fdff3311197"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:35:31.299677 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:31.299657 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d2ee41-aaf6-46fe-8c0a-1fdff3311197-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:31.301283 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:31.301256 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3d2ee41-aaf6-46fe-8c0a-1fdff3311197-metrics-certs" pod "b3d2ee41-aaf6-46fe-8c0a-1fdff3311197" (UID: "b3d2ee41-aaf6-46fe-8c0a-1fdff3311197"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:35:31.301390 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:31.301295 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3d2ee41-aaf6-46fe-8c0a-1fdff3311197-kube-api-access-vhcx5" pod "b3d2ee41-aaf6-46fe-8c0a-1fdff3311197" (UID: "b3d2ee41-aaf6-46fe-8c0a-1fdff3311197"). InnerVolumeSpecName "kube-api-access-vhcx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:35:31.301390 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:31.301315 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3d2ee41-aaf6-46fe-8c0a-1fdff3311197-maas-api-tls" pod "b3d2ee41-aaf6-46fe-8c0a-1fdff3311197" (UID: "b3d2ee41-aaf6-46fe-8c0a-1fdff3311197"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:35:31.413061 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:31.413021 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3d2ee41-aaf6-46fe-8c0a-1fdff3311197-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:31.413061 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:31.413064 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhcx5\" (UniqueName: \"kubernetes.io/projected/b3d2ee41-aaf6-46fe-8c0a-1fdff3311197-kube-api-access-vhcx5\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:31.413272 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:31.413083 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3d2ee41-aaf6-46fe-8c0a-1fdff3311197-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:31.477075 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:31.476991 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-7f5cc8bc5-q4q99"] Nov 02 23:35:31.477256 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:31.477206 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-7f5cc8bc5-q4q99" podUID="41f08c9c-6fe4-4a4d-a633-9dce1493b831" containerName="payload-pre-processing" containerID="cri-o://810ee7a4f8c98c00bcfe6ed479ac1b7972563823d298615a58eceab32ef24f8d" gracePeriod=30 Nov 02 23:35:31.731789 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:31.731731 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-7f5cc8bc5-q4q99" Nov 02 23:35:31.816563 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:31.816526 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41f08c9c-6fe4-4a4d-a633-9dce1493b831-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41f08c9c-6fe4-4a4d-a633-9dce1493b831-plugins-config-volume\") pod \"41f08c9c-6fe4-4a4d-a633-9dce1493b831\" (UID: \"41f08c9c-6fe4-4a4d-a633-9dce1493b831\") " Nov 02 23:35:31.816757 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:31.816718 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41f08c9c-6fe4-4a4d-a633-9dce1493b831-kube-api-access-wk7dh\" (UniqueName: \"kubernetes.io/projected/41f08c9c-6fe4-4a4d-a633-9dce1493b831-kube-api-access-wk7dh\") pod \"41f08c9c-6fe4-4a4d-a633-9dce1493b831\" (UID: \"41f08c9c-6fe4-4a4d-a633-9dce1493b831\") " Nov 02 23:35:31.817005 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:31.816970 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41f08c9c-6fe4-4a4d-a633-9dce1493b831-plugins-config-volume" pod "41f08c9c-6fe4-4a4d-a633-9dce1493b831" (UID: "41f08c9c-6fe4-4a4d-a633-9dce1493b831"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:35:31.817227 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:31.817205 5076 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41f08c9c-6fe4-4a4d-a633-9dce1493b831-plugins-config-volume\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:31.819003 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:31.818979 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41f08c9c-6fe4-4a4d-a633-9dce1493b831-kube-api-access-wk7dh" pod "41f08c9c-6fe4-4a4d-a633-9dce1493b831" (UID: "41f08c9c-6fe4-4a4d-a633-9dce1493b831"). InnerVolumeSpecName "kube-api-access-wk7dh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:35:31.918525 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:31.918487 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk7dh\" (UniqueName: \"kubernetes.io/projected/41f08c9c-6fe4-4a4d-a633-9dce1493b831-kube-api-access-wk7dh\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:32.124226 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:32.124190 5076 generic.go:356] "Generic (PLEG): container finished" podID="41f08c9c-6fe4-4a4d-a633-9dce1493b831" containerID="810ee7a4f8c98c00bcfe6ed479ac1b7972563823d298615a58eceab32ef24f8d" exitCode=0 Nov 02 23:35:32.124697 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:32.124241 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-7f5cc8bc5-q4q99" Nov 02 23:35:32.124697 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:32.124269 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-7f5cc8bc5-q4q99" event={"ID":"41f08c9c-6fe4-4a4d-a633-9dce1493b831","Type":"ContainerDied","Data":"810ee7a4f8c98c00bcfe6ed479ac1b7972563823d298615a58eceab32ef24f8d"} Nov 02 23:35:32.124697 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:32.124321 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-7f5cc8bc5-q4q99" event={"ID":"41f08c9c-6fe4-4a4d-a633-9dce1493b831","Type":"ContainerDied","Data":"813faf094b3c546ade77812b631b4c3d39cca77d6f284e87c03c7acf4c878db5"} Nov 02 23:35:32.124697 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:32.124343 5076 scope.go:122] "RemoveContainer" containerID="810ee7a4f8c98c00bcfe6ed479ac1b7972563823d298615a58eceab32ef24f8d" Nov 02 23:35:32.125801 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:32.125782 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94" event={"ID":"b3d2ee41-aaf6-46fe-8c0a-1fdff3311197","Type":"ContainerDied","Data":"6754eaa04ee8ad56723b9e304ed9f0ac3281ec9a89e6ec67e307462b61edcb18"} Nov 02 23:35:32.125801 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:32.125796 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94" Nov 02 23:35:32.135260 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:32.135238 5076 scope.go:122] "RemoveContainer" containerID="810ee7a4f8c98c00bcfe6ed479ac1b7972563823d298615a58eceab32ef24f8d" Nov 02 23:35:32.135992 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:35:32.135965 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"810ee7a4f8c98c00bcfe6ed479ac1b7972563823d298615a58eceab32ef24f8d\": container with ID starting with 810ee7a4f8c98c00bcfe6ed479ac1b7972563823d298615a58eceab32ef24f8d not found: ID does not exist" containerID="810ee7a4f8c98c00bcfe6ed479ac1b7972563823d298615a58eceab32ef24f8d" Nov 02 23:35:32.136107 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:32.135995 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"810ee7a4f8c98c00bcfe6ed479ac1b7972563823d298615a58eceab32ef24f8d"} err="failed to get container status \"810ee7a4f8c98c00bcfe6ed479ac1b7972563823d298615a58eceab32ef24f8d\": rpc error: code = NotFound desc = could not find container \"810ee7a4f8c98c00bcfe6ed479ac1b7972563823d298615a58eceab32ef24f8d\": container with ID starting with 810ee7a4f8c98c00bcfe6ed479ac1b7972563823d298615a58eceab32ef24f8d not found: ID does not exist" Nov 02 23:35:32.136107 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:32.136014 5076 scope.go:122] "RemoveContainer" containerID="e7297e705d87bac61193ce57ce37d9dcdbd515b602754e9da23b7952795ee977" Nov 02 23:35:32.162905 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:32.162870 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z"] Nov 02 23:35:32.163103 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:32.163069 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z" podUID="6ac3a7a9-9008-4869-8067-c9e82ad9a79f" containerName="maas-api" containerID="cri-o://1e2a3fb83bebb99b18716f99dcb4b984e0f2ec9f376c7fa9a5f7ffd1c4a8b399" gracePeriod=30 Nov 02 23:35:32.281304 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:32.281120 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94"] Nov 02 23:35:32.285021 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:32.284997 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a3909-79695669c8-8sq94"] Nov 02 23:35:32.468467 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:32.468444 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z" Nov 02 23:35:32.504033 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:32.503996 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-7f5cc8bc5-q4q99"] Nov 02 23:35:32.506589 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:32.506567 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-7f5cc8bc5-q4q99"] Nov 02 23:35:32.526240 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:32.526214 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ac3a7a9-9008-4869-8067-c9e82ad9a79f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ac3a7a9-9008-4869-8067-c9e82ad9a79f-metrics-certs\") pod \"6ac3a7a9-9008-4869-8067-c9e82ad9a79f\" (UID: \"6ac3a7a9-9008-4869-8067-c9e82ad9a79f\") " Nov 02 23:35:32.526374 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:32.526358 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ac3a7a9-9008-4869-8067-c9e82ad9a79f-kube-api-access-lc4hl\" (UniqueName: \"kubernetes.io/projected/6ac3a7a9-9008-4869-8067-c9e82ad9a79f-kube-api-access-lc4hl\") pod \"6ac3a7a9-9008-4869-8067-c9e82ad9a79f\" (UID: \"6ac3a7a9-9008-4869-8067-c9e82ad9a79f\") " Nov 02 23:35:32.526435 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:32.526421 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ac3a7a9-9008-4869-8067-c9e82ad9a79f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ac3a7a9-9008-4869-8067-c9e82ad9a79f-openshift-service-ca\") pod \"6ac3a7a9-9008-4869-8067-c9e82ad9a79f\" (UID: \"6ac3a7a9-9008-4869-8067-c9e82ad9a79f\") " Nov 02 23:35:32.526491 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:32.526480 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ac3a7a9-9008-4869-8067-c9e82ad9a79f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ac3a7a9-9008-4869-8067-c9e82ad9a79f-maas-api-tls\") pod \"6ac3a7a9-9008-4869-8067-c9e82ad9a79f\" (UID: \"6ac3a7a9-9008-4869-8067-c9e82ad9a79f\") " Nov 02 23:35:32.526871 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:32.526843 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ac3a7a9-9008-4869-8067-c9e82ad9a79f-openshift-service-ca" pod "6ac3a7a9-9008-4869-8067-c9e82ad9a79f" (UID: "6ac3a7a9-9008-4869-8067-c9e82ad9a79f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:35:32.527040 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:32.526969 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ac3a7a9-9008-4869-8067-c9e82ad9a79f-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:32.528449 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:32.528426 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ac3a7a9-9008-4869-8067-c9e82ad9a79f-metrics-certs" pod "6ac3a7a9-9008-4869-8067-c9e82ad9a79f" (UID: "6ac3a7a9-9008-4869-8067-c9e82ad9a79f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:35:32.528539 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:32.528482 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ac3a7a9-9008-4869-8067-c9e82ad9a79f-maas-api-tls" pod "6ac3a7a9-9008-4869-8067-c9e82ad9a79f" (UID: "6ac3a7a9-9008-4869-8067-c9e82ad9a79f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:35:32.528539 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:32.528514 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ac3a7a9-9008-4869-8067-c9e82ad9a79f-kube-api-access-lc4hl" pod "6ac3a7a9-9008-4869-8067-c9e82ad9a79f" (UID: "6ac3a7a9-9008-4869-8067-c9e82ad9a79f"). InnerVolumeSpecName "kube-api-access-lc4hl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:35:32.628620 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:32.628583 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc4hl\" (UniqueName: \"kubernetes.io/projected/6ac3a7a9-9008-4869-8067-c9e82ad9a79f-kube-api-access-lc4hl\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:32.628620 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:32.628622 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ac3a7a9-9008-4869-8067-c9e82ad9a79f-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:32.628864 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:32.628637 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ac3a7a9-9008-4869-8067-c9e82ad9a79f-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:33.171916 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:33.171440 5076 generic.go:356] "Generic (PLEG): container finished" podID="6ac3a7a9-9008-4869-8067-c9e82ad9a79f" containerID="1e2a3fb83bebb99b18716f99dcb4b984e0f2ec9f376c7fa9a5f7ffd1c4a8b399" exitCode=0 Nov 02 23:35:33.171916 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:33.171606 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z" Nov 02 23:35:33.178086 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:33.178046 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41f08c9c-6fe4-4a4d-a633-9dce1493b831" path="/var/lib/kubelet/pods/41f08c9c-6fe4-4a4d-a633-9dce1493b831/volumes" Nov 02 23:35:33.178899 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:33.178858 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3d2ee41-aaf6-46fe-8c0a-1fdff3311197" path="/var/lib/kubelet/pods/b3d2ee41-aaf6-46fe-8c0a-1fdff3311197/volumes" Nov 02 23:35:33.179706 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:33.179682 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z" event={"ID":"6ac3a7a9-9008-4869-8067-c9e82ad9a79f","Type":"ContainerDied","Data":"1e2a3fb83bebb99b18716f99dcb4b984e0f2ec9f376c7fa9a5f7ffd1c4a8b399"} Nov 02 23:35:33.179805 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:33.179724 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z" event={"ID":"6ac3a7a9-9008-4869-8067-c9e82ad9a79f","Type":"ContainerDied","Data":"1b063c3e49ed7806e54d9fb4866d1abf6affe32122dfc6607b2d34f2060b2294"} Nov 02 23:35:33.179805 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:33.179758 5076 scope.go:122] "RemoveContainer" containerID="1e2a3fb83bebb99b18716f99dcb4b984e0f2ec9f376c7fa9a5f7ffd1c4a8b399" Nov 02 23:35:33.200980 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:33.200954 5076 scope.go:122] "RemoveContainer" containerID="1e2a3fb83bebb99b18716f99dcb4b984e0f2ec9f376c7fa9a5f7ffd1c4a8b399" Nov 02 23:35:33.201412 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:35:33.201384 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e2a3fb83bebb99b18716f99dcb4b984e0f2ec9f376c7fa9a5f7ffd1c4a8b399\": container with ID starting with 1e2a3fb83bebb99b18716f99dcb4b984e0f2ec9f376c7fa9a5f7ffd1c4a8b399 not found: ID does not exist" containerID="1e2a3fb83bebb99b18716f99dcb4b984e0f2ec9f376c7fa9a5f7ffd1c4a8b399" Nov 02 23:35:33.201536 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:33.201421 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e2a3fb83bebb99b18716f99dcb4b984e0f2ec9f376c7fa9a5f7ffd1c4a8b399"} err="failed to get container status \"1e2a3fb83bebb99b18716f99dcb4b984e0f2ec9f376c7fa9a5f7ffd1c4a8b399\": rpc error: code = NotFound desc = could not find container \"1e2a3fb83bebb99b18716f99dcb4b984e0f2ec9f376c7fa9a5f7ffd1c4a8b399\": container with ID starting with 1e2a3fb83bebb99b18716f99dcb4b984e0f2ec9f376c7fa9a5f7ffd1c4a8b399 not found: ID does not exist" Nov 02 23:35:33.369818 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:33.369783 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z"] Nov 02 23:35:33.382934 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:33.382905 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-fb5fa2-86d88d8ff5-9xr6z"] Nov 02 23:35:35.148266 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:35.148229 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ac3a7a9-9008-4869-8067-c9e82ad9a79f" path="/var/lib/kubelet/pods/6ac3a7a9-9008-4869-8067-c9e82ad9a79f/volumes" Nov 02 23:35:39.884072 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:35:39.884033 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:35:44.310206 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:44.310168 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj"] Nov 02 23:35:44.310720 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:44.310428 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" podUID="e8d06dd9-47b5-442a-ad96-f8616e6570a8" containerName="istio-proxy" containerID="cri-o://f470c2f29506f4d1d48af594511abf37620bccdc141ad07f1f43255ff8aa1aec" gracePeriod=30 Nov 02 23:35:45.211448 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:45.211413 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-b9756748c-pqkdd"] Nov 02 23:35:45.212023 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:45.212006 5076 cpu_manager.go:414] "containerMap: removing container" podUID="b3d2ee41-aaf6-46fe-8c0a-1fdff3311197" containerName="maas-api" Nov 02 23:35:45.212023 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:45.212022 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3d2ee41-aaf6-46fe-8c0a-1fdff3311197" containerName="maas-api" Nov 02 23:35:45.212180 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:45.212044 5076 cpu_manager.go:414] "containerMap: removing container" podUID="41f08c9c-6fe4-4a4d-a633-9dce1493b831" containerName="payload-pre-processing" Nov 02 23:35:45.212180 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:45.212052 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41f08c9c-6fe4-4a4d-a633-9dce1493b831" containerName="payload-pre-processing" Nov 02 23:35:45.212180 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:45.212087 5076 cpu_manager.go:414] "containerMap: removing container" podUID="6ac3a7a9-9008-4869-8067-c9e82ad9a79f" containerName="maas-api" Nov 02 23:35:45.212180 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:45.212094 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ac3a7a9-9008-4869-8067-c9e82ad9a79f" containerName="maas-api" Nov 02 23:35:45.212337 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:45.212217 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ac3a7a9-9008-4869-8067-c9e82ad9a79f" containerName="maas-api" Nov 02 23:35:45.212337 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:45.212239 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3d2ee41-aaf6-46fe-8c0a-1fdff3311197" containerName="maas-api" Nov 02 23:35:45.212337 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:45.212255 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="41f08c9c-6fe4-4a4d-a633-9dce1493b831" containerName="payload-pre-processing" Nov 02 23:35:45.217197 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:45.217178 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-b9756748c-pqkdd" Nov 02 23:35:45.267096 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:45.267060 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-db5d87\"" Nov 02 23:35:45.267425 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:45.267401 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-b9756748c-pqkdd"] Nov 02 23:35:45.311710 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:45.311678 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-db5d87-dockercfg-dkhm9\"" Nov 02 23:35:45.319365 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:45.319337 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm68l\" (UniqueName: \"kubernetes.io/projected/8895d1e5-9514-4125-8343-e8d3c9cf3d09-kube-api-access-mm68l\") pod \"payload-pre-processing-e2e-ipp-b-db5d87-b9756748c-pqkdd\" (UID: \"8895d1e5-9514-4125-8343-e8d3c9cf3d09\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-b9756748c-pqkdd" Nov 02 23:35:45.319481 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:45.319387 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8895d1e5-9514-4125-8343-e8d3c9cf3d09-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-db5d87-b9756748c-pqkdd\" (UID: \"8895d1e5-9514-4125-8343-e8d3c9cf3d09\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-b9756748c-pqkdd" Nov 02 23:35:45.420401 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:45.420367 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm68l\" (UniqueName: \"kubernetes.io/projected/8895d1e5-9514-4125-8343-e8d3c9cf3d09-kube-api-access-mm68l\") pod \"payload-pre-processing-e2e-ipp-b-db5d87-b9756748c-pqkdd\" (UID: \"8895d1e5-9514-4125-8343-e8d3c9cf3d09\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-b9756748c-pqkdd" Nov 02 23:35:45.420600 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:45.420416 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8895d1e5-9514-4125-8343-e8d3c9cf3d09-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-db5d87-b9756748c-pqkdd\" (UID: \"8895d1e5-9514-4125-8343-e8d3c9cf3d09\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-b9756748c-pqkdd" Nov 02 23:35:45.421145 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:45.421122 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8895d1e5-9514-4125-8343-e8d3c9cf3d09-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-db5d87-b9756748c-pqkdd\" (UID: \"8895d1e5-9514-4125-8343-e8d3c9cf3d09\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-b9756748c-pqkdd" Nov 02 23:35:45.474834 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:45.474759 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm68l\" (UniqueName: \"kubernetes.io/projected/8895d1e5-9514-4125-8343-e8d3c9cf3d09-kube-api-access-mm68l\") pod \"payload-pre-processing-e2e-ipp-b-db5d87-b9756748c-pqkdd\" (UID: \"8895d1e5-9514-4125-8343-e8d3c9cf3d09\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-b9756748c-pqkdd" Nov 02 23:35:45.531278 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:45.531237 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-b9756748c-pqkdd" Nov 02 23:35:45.864781 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:45.864699 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db5d87-598f6cbf4d-2m2xw"] Nov 02 23:35:45.876323 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:45.876291 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5d87-598f6cbf4d-2m2xw" Nov 02 23:35:45.898669 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:45.898643 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-b9756748c-pqkdd"] Nov 02 23:35:45.899410 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:35:45.899383 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8895d1e5_9514_4125_8343_e8d3c9cf3d09.slice/crio-0873c2e81bb6b2ea7f54e16d8967c1c6436417e24edecf86373eb7faeca50037 WatchSource:0}: Error finding container 0873c2e81bb6b2ea7f54e16d8967c1c6436417e24edecf86373eb7faeca50037: Status 404 returned error can't find the container with id 0873c2e81bb6b2ea7f54e16d8967c1c6436417e24edecf86373eb7faeca50037 Nov 02 23:35:45.922366 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:45.922342 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db5d87-598f6cbf4d-2m2xw"] Nov 02 23:35:46.028103 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:46.028064 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjdzh\" (UniqueName: \"kubernetes.io/projected/e02e672a-a973-4706-9c2b-a307aabafa9b-kube-api-access-jjdzh\") pod \"payload-processing-e2e-ipp-b-db5d87-598f6cbf4d-2m2xw\" (UID: \"e02e672a-a973-4706-9c2b-a307aabafa9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db5d87-598f6cbf4d-2m2xw" Nov 02 23:35:46.028276 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:46.028178 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e02e672a-a973-4706-9c2b-a307aabafa9b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-db5d87-598f6cbf4d-2m2xw\" (UID: \"e02e672a-a973-4706-9c2b-a307aabafa9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db5d87-598f6cbf4d-2m2xw" Nov 02 23:35:46.129290 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:46.129205 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjdzh\" (UniqueName: \"kubernetes.io/projected/e02e672a-a973-4706-9c2b-a307aabafa9b-kube-api-access-jjdzh\") pod \"payload-processing-e2e-ipp-b-db5d87-598f6cbf4d-2m2xw\" (UID: \"e02e672a-a973-4706-9c2b-a307aabafa9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db5d87-598f6cbf4d-2m2xw" Nov 02 23:35:46.129290 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:46.129272 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e02e672a-a973-4706-9c2b-a307aabafa9b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-db5d87-598f6cbf4d-2m2xw\" (UID: \"e02e672a-a973-4706-9c2b-a307aabafa9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db5d87-598f6cbf4d-2m2xw" Nov 02 23:35:46.130024 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:46.130004 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e02e672a-a973-4706-9c2b-a307aabafa9b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-db5d87-598f6cbf4d-2m2xw\" (UID: \"e02e672a-a973-4706-9c2b-a307aabafa9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db5d87-598f6cbf4d-2m2xw" Nov 02 23:35:46.183422 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:46.183393 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjdzh\" (UniqueName: \"kubernetes.io/projected/e02e672a-a973-4706-9c2b-a307aabafa9b-kube-api-access-jjdzh\") pod \"payload-processing-e2e-ipp-b-db5d87-598f6cbf4d-2m2xw\" (UID: \"e02e672a-a973-4706-9c2b-a307aabafa9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db5d87-598f6cbf4d-2m2xw" Nov 02 23:35:46.191267 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:46.191243 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5d87-598f6cbf4d-2m2xw" Nov 02 23:35:46.291042 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:46.290937 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-b9756748c-pqkdd" event={"ID":"8895d1e5-9514-4125-8343-e8d3c9cf3d09","Type":"ContainerStarted","Data":"61db32bc96293dafdb77b3503283337e84c28f3f636fc48fa2450caf135b5962"} Nov 02 23:35:46.291042 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:46.291003 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-b9756748c-pqkdd" event={"ID":"8895d1e5-9514-4125-8343-e8d3c9cf3d09","Type":"ContainerStarted","Data":"0873c2e81bb6b2ea7f54e16d8967c1c6436417e24edecf86373eb7faeca50037"} Nov 02 23:35:46.291292 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:46.291067 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-b9756748c-pqkdd" Nov 02 23:35:46.401232 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:46.401133 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-b9756748c-pqkdd" podStartSLOduration=1.401118912 podStartE2EDuration="1.401118912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:35:46.398348748 +0000 UTC m=+2595.366880534" watchObservedRunningTime="2026-11-02 23:35:46.401118912 +0000 UTC m=+2595.369650695" Nov 02 23:35:46.556871 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:46.556842 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db5d87-598f6cbf4d-2m2xw"] Nov 02 23:35:46.558429 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:35:46.558400 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode02e672a_a973_4706_9c2b_a307aabafa9b.slice/crio-e5fae53e3afd6c92f045f94679138ecd3d4b147df3fe6be990b60a4663a9fc26 WatchSource:0}: Error finding container e5fae53e3afd6c92f045f94679138ecd3d4b147df3fe6be990b60a4663a9fc26: Status 404 returned error can't find the container with id e5fae53e3afd6c92f045f94679138ecd3d4b147df3fe6be990b60a4663a9fc26 Nov 02 23:35:47.300802 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:47.300762 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5d87-598f6cbf4d-2m2xw" event={"ID":"e02e672a-a973-4706-9c2b-a307aabafa9b","Type":"ContainerStarted","Data":"4ce532c18fa6066c4db7ad370392691aade147378c650c741472922e7ebc864b"} Nov 02 23:35:47.301020 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:47.300823 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5d87-598f6cbf4d-2m2xw" event={"ID":"e02e672a-a973-4706-9c2b-a307aabafa9b","Type":"ContainerStarted","Data":"e5fae53e3afd6c92f045f94679138ecd3d4b147df3fe6be990b60a4663a9fc26"} Nov 02 23:35:47.301092 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:47.301029 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5d87-598f6cbf4d-2m2xw" Nov 02 23:35:47.412611 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:47.412550 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5d87-598f6cbf4d-2m2xw" podStartSLOduration=2.412530914 podStartE2EDuration="2.412530914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:35:47.409966448 +0000 UTC m=+2596.378498246" watchObservedRunningTime="2026-11-02 23:35:47.412530914 +0000 UTC m=+2596.381062701" Nov 02 23:35:49.614903 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:49.614872 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" Nov 02 23:35:49.785158 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:49.785121 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8d06dd9-47b5-442a-ad96-f8616e6570a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8d06dd9-47b5-442a-ad96-f8616e6570a8-istio-data\") pod \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\" (UID: \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\") " Nov 02 23:35:49.785338 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:49.785204 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8d06dd9-47b5-442a-ad96-f8616e6570a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8d06dd9-47b5-442a-ad96-f8616e6570a8-istio-podinfo\") pod \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\" (UID: \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\") " Nov 02 23:35:49.785338 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:49.785267 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8d06dd9-47b5-442a-ad96-f8616e6570a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8d06dd9-47b5-442a-ad96-f8616e6570a8-istio-envoy\") pod \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\" (UID: \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\") " Nov 02 23:35:49.785338 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:49.785310 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8d06dd9-47b5-442a-ad96-f8616e6570a8-kube-api-access-txjw8\" (UniqueName: \"kubernetes.io/projected/e8d06dd9-47b5-442a-ad96-f8616e6570a8-kube-api-access-txjw8\") pod \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\" (UID: \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\") " Nov 02 23:35:49.785520 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:49.785378 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8d06dd9-47b5-442a-ad96-f8616e6570a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8d06dd9-47b5-442a-ad96-f8616e6570a8-istiod-ca-cert\") pod \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\" (UID: \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\") " Nov 02 23:35:49.785520 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:49.785429 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8d06dd9-47b5-442a-ad96-f8616e6570a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8d06dd9-47b5-442a-ad96-f8616e6570a8-workload-socket\") pod \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\" (UID: \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\") " Nov 02 23:35:49.785520 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:49.785461 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8d06dd9-47b5-442a-ad96-f8616e6570a8-istio-token\" (UniqueName: \"kubernetes.io/projected/e8d06dd9-47b5-442a-ad96-f8616e6570a8-istio-token\") pod \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\" (UID: \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\") " Nov 02 23:35:49.785520 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:49.785466 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8d06dd9-47b5-442a-ad96-f8616e6570a8-istio-data" pod "e8d06dd9-47b5-442a-ad96-f8616e6570a8" (UID: "e8d06dd9-47b5-442a-ad96-f8616e6570a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:35:49.785520 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:49.785506 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8d06dd9-47b5-442a-ad96-f8616e6570a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8d06dd9-47b5-442a-ad96-f8616e6570a8-workload-certs\") pod \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\" (UID: \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\") " Nov 02 23:35:49.785827 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:49.785611 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8d06dd9-47b5-442a-ad96-f8616e6570a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8d06dd9-47b5-442a-ad96-f8616e6570a8-credential-socket\") pod \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\" (UID: \"e8d06dd9-47b5-442a-ad96-f8616e6570a8\") " Nov 02 23:35:49.785827 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:49.785799 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8d06dd9-47b5-442a-ad96-f8616e6570a8-istiod-ca-cert" pod "e8d06dd9-47b5-442a-ad96-f8616e6570a8" (UID: "e8d06dd9-47b5-442a-ad96-f8616e6570a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:35:49.786226 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:49.786186 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8d06dd9-47b5-442a-ad96-f8616e6570a8-workload-certs" pod "e8d06dd9-47b5-442a-ad96-f8616e6570a8" (UID: "e8d06dd9-47b5-442a-ad96-f8616e6570a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:35:49.786469 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:49.786434 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8d06dd9-47b5-442a-ad96-f8616e6570a8-credential-socket" pod "e8d06dd9-47b5-442a-ad96-f8616e6570a8" (UID: "e8d06dd9-47b5-442a-ad96-f8616e6570a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:35:49.787545 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:49.786619 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8d06dd9-47b5-442a-ad96-f8616e6570a8-workload-socket" pod "e8d06dd9-47b5-442a-ad96-f8616e6570a8" (UID: "e8d06dd9-47b5-442a-ad96-f8616e6570a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:35:49.788162 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:49.788143 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8d06dd9-47b5-442a-ad96-f8616e6570a8-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:49.788311 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:49.788295 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8d06dd9-47b5-442a-ad96-f8616e6570a8-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:49.788449 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:49.788430 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8d06dd9-47b5-442a-ad96-f8616e6570a8-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:49.788555 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:49.788451 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8d06dd9-47b5-442a-ad96-f8616e6570a8-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:49.788555 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:49.788466 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8d06dd9-47b5-442a-ad96-f8616e6570a8-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:49.788555 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:49.788199 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8d06dd9-47b5-442a-ad96-f8616e6570a8-istio-podinfo" pod "e8d06dd9-47b5-442a-ad96-f8616e6570a8" (UID: "e8d06dd9-47b5-442a-ad96-f8616e6570a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:35:49.788555 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:49.788489 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8d06dd9-47b5-442a-ad96-f8616e6570a8-istio-token" pod "e8d06dd9-47b5-442a-ad96-f8616e6570a8" (UID: "e8d06dd9-47b5-442a-ad96-f8616e6570a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:35:49.788757 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:49.788657 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8d06dd9-47b5-442a-ad96-f8616e6570a8-istio-envoy" pod "e8d06dd9-47b5-442a-ad96-f8616e6570a8" (UID: "e8d06dd9-47b5-442a-ad96-f8616e6570a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:35:49.789548 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:49.789523 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8d06dd9-47b5-442a-ad96-f8616e6570a8-kube-api-access-txjw8" pod "e8d06dd9-47b5-442a-ad96-f8616e6570a8" (UID: "e8d06dd9-47b5-442a-ad96-f8616e6570a8"). InnerVolumeSpecName "kube-api-access-txjw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:35:49.890210 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:49.890136 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8d06dd9-47b5-442a-ad96-f8616e6570a8-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:49.890210 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:49.890160 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8d06dd9-47b5-442a-ad96-f8616e6570a8-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:49.890210 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:49.890171 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txjw8\" (UniqueName: \"kubernetes.io/projected/e8d06dd9-47b5-442a-ad96-f8616e6570a8-kube-api-access-txjw8\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:49.890210 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:49.890183 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8d06dd9-47b5-442a-ad96-f8616e6570a8-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:50.319678 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.319644 5076 generic.go:356] "Generic (PLEG): container finished" podID="e8d06dd9-47b5-442a-ad96-f8616e6570a8" containerID="f470c2f29506f4d1d48af594511abf37620bccdc141ad07f1f43255ff8aa1aec" exitCode=0 Nov 02 23:35:50.319991 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.319701 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" event={"ID":"e8d06dd9-47b5-442a-ad96-f8616e6570a8","Type":"ContainerDied","Data":"f470c2f29506f4d1d48af594511abf37620bccdc141ad07f1f43255ff8aa1aec"} Nov 02 23:35:50.319991 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.319705 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" Nov 02 23:35:50.319991 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.319753 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj" event={"ID":"e8d06dd9-47b5-442a-ad96-f8616e6570a8","Type":"ContainerDied","Data":"e905672868ea0e82b42082637973b2229f6f78d8ab23848a83409800141276ee"} Nov 02 23:35:50.319991 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.319782 5076 scope.go:122] "RemoveContainer" containerID="f470c2f29506f4d1d48af594511abf37620bccdc141ad07f1f43255ff8aa1aec" Nov 02 23:35:50.330557 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.330540 5076 scope.go:122] "RemoveContainer" containerID="f01d415c678dd1f861af35a82f6eb8c5b5bb378a8770c175d19b4360b25ff7d5" Nov 02 23:35:50.339955 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.339936 5076 scope.go:122] "RemoveContainer" containerID="f470c2f29506f4d1d48af594511abf37620bccdc141ad07f1f43255ff8aa1aec" Nov 02 23:35:50.340192 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:35:50.340174 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f470c2f29506f4d1d48af594511abf37620bccdc141ad07f1f43255ff8aa1aec\": container with ID starting with f470c2f29506f4d1d48af594511abf37620bccdc141ad07f1f43255ff8aa1aec not found: ID does not exist" containerID="f470c2f29506f4d1d48af594511abf37620bccdc141ad07f1f43255ff8aa1aec" Nov 02 23:35:50.340243 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.340201 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f470c2f29506f4d1d48af594511abf37620bccdc141ad07f1f43255ff8aa1aec"} err="failed to get container status \"f470c2f29506f4d1d48af594511abf37620bccdc141ad07f1f43255ff8aa1aec\": rpc error: code = NotFound desc = could not find container \"f470c2f29506f4d1d48af594511abf37620bccdc141ad07f1f43255ff8aa1aec\": container with ID starting with f470c2f29506f4d1d48af594511abf37620bccdc141ad07f1f43255ff8aa1aec not found: ID does not exist" Nov 02 23:35:50.340243 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.340218 5076 scope.go:122] "RemoveContainer" containerID="f01d415c678dd1f861af35a82f6eb8c5b5bb378a8770c175d19b4360b25ff7d5" Nov 02 23:35:50.340416 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:35:50.340400 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f01d415c678dd1f861af35a82f6eb8c5b5bb378a8770c175d19b4360b25ff7d5\": container with ID starting with f01d415c678dd1f861af35a82f6eb8c5b5bb378a8770c175d19b4360b25ff7d5 not found: ID does not exist" containerID="f01d415c678dd1f861af35a82f6eb8c5b5bb378a8770c175d19b4360b25ff7d5" Nov 02 23:35:50.340456 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.340421 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f01d415c678dd1f861af35a82f6eb8c5b5bb378a8770c175d19b4360b25ff7d5"} err="failed to get container status \"f01d415c678dd1f861af35a82f6eb8c5b5bb378a8770c175d19b4360b25ff7d5\": rpc error: code = NotFound desc = could not find container \"f01d415c678dd1f861af35a82f6eb8c5b5bb378a8770c175d19b4360b25ff7d5\": container with ID starting with f01d415c678dd1f861af35a82f6eb8c5b5bb378a8770c175d19b4360b25ff7d5 not found: ID does not exist" Nov 02 23:35:50.410929 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.410880 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5"] Nov 02 23:35:50.411487 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.411473 5076 cpu_manager.go:414] "containerMap: removing container" podUID="e8d06dd9-47b5-442a-ad96-f8616e6570a8" containerName="istio-proxy" Nov 02 23:35:50.411557 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.411488 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8d06dd9-47b5-442a-ad96-f8616e6570a8" containerName="istio-proxy" Nov 02 23:35:50.411557 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.411512 5076 cpu_manager.go:414] "containerMap: removing container" podUID="e8d06dd9-47b5-442a-ad96-f8616e6570a8" containerName="istio-proxy" Nov 02 23:35:50.411557 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.411517 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8d06dd9-47b5-442a-ad96-f8616e6570a8" containerName="istio-proxy" Nov 02 23:35:50.411650 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.411599 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8d06dd9-47b5-442a-ad96-f8616e6570a8" containerName="istio-proxy" Nov 02 23:35:50.411650 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.411611 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8d06dd9-47b5-442a-ad96-f8616e6570a8" containerName="istio-proxy" Nov 02 23:35:50.416609 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.416587 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" Nov 02 23:35:50.437467 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.437439 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89"] Nov 02 23:35:50.441540 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.441522 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" Nov 02 23:35:50.476970 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.476938 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj"] Nov 02 23:35:50.481288 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.481261 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-fb5fa2-openshift-default-54d9c689cc-8ptkj"] Nov 02 23:35:50.508886 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.508856 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5"] Nov 02 23:35:50.559328 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.559290 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89"] Nov 02 23:35:50.597703 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.597613 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crmh9\" (UniqueName: \"kubernetes.io/projected/c9975a71-d883-4d38-8b69-e3aa21574b13-kube-api-access-crmh9\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89\" (UID: \"c9975a71-d883-4d38-8b69-e3aa21574b13\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" Nov 02 23:35:50.597703 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.597654 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00d606e8-8e81-4b50-884b-d4e6bb36659c-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5\" (UID: \"00d606e8-8e81-4b50-884b-d4e6bb36659c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" Nov 02 23:35:50.597979 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.597719 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00d606e8-8e81-4b50-884b-d4e6bb36659c-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5\" (UID: \"00d606e8-8e81-4b50-884b-d4e6bb36659c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" Nov 02 23:35:50.597979 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.597771 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9975a71-d883-4d38-8b69-e3aa21574b13-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89\" (UID: \"c9975a71-d883-4d38-8b69-e3aa21574b13\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" Nov 02 23:35:50.597979 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.597863 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9975a71-d883-4d38-8b69-e3aa21574b13-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89\" (UID: \"c9975a71-d883-4d38-8b69-e3aa21574b13\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" Nov 02 23:35:50.597979 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.597926 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9975a71-d883-4d38-8b69-e3aa21574b13-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89\" (UID: \"c9975a71-d883-4d38-8b69-e3aa21574b13\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" Nov 02 23:35:50.598149 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.597981 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00d606e8-8e81-4b50-884b-d4e6bb36659c-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5\" (UID: \"00d606e8-8e81-4b50-884b-d4e6bb36659c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" Nov 02 23:35:50.598149 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.598029 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d606e8-8e81-4b50-884b-d4e6bb36659c-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5\" (UID: \"00d606e8-8e81-4b50-884b-d4e6bb36659c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" Nov 02 23:35:50.598149 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.598063 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9975a71-d883-4d38-8b69-e3aa21574b13-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89\" (UID: \"c9975a71-d883-4d38-8b69-e3aa21574b13\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" Nov 02 23:35:50.598149 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.598138 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9975a71-d883-4d38-8b69-e3aa21574b13-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89\" (UID: \"c9975a71-d883-4d38-8b69-e3aa21574b13\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" Nov 02 23:35:50.598298 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.598174 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00d606e8-8e81-4b50-884b-d4e6bb36659c-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5\" (UID: \"00d606e8-8e81-4b50-884b-d4e6bb36659c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" Nov 02 23:35:50.598298 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.598206 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d606e8-8e81-4b50-884b-d4e6bb36659c-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5\" (UID: \"00d606e8-8e81-4b50-884b-d4e6bb36659c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" Nov 02 23:35:50.598298 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.598269 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00d606e8-8e81-4b50-884b-d4e6bb36659c-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5\" (UID: \"00d606e8-8e81-4b50-884b-d4e6bb36659c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" Nov 02 23:35:50.598401 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.598314 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00d606e8-8e81-4b50-884b-d4e6bb36659c-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5\" (UID: \"00d606e8-8e81-4b50-884b-d4e6bb36659c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" Nov 02 23:35:50.598401 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.598358 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9975a71-d883-4d38-8b69-e3aa21574b13-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89\" (UID: \"c9975a71-d883-4d38-8b69-e3aa21574b13\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" Nov 02 23:35:50.598479 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.598400 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29r8f\" (UniqueName: \"kubernetes.io/projected/00d606e8-8e81-4b50-884b-d4e6bb36659c-kube-api-access-29r8f\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5\" (UID: \"00d606e8-8e81-4b50-884b-d4e6bb36659c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" Nov 02 23:35:50.598479 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.598421 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9975a71-d883-4d38-8b69-e3aa21574b13-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89\" (UID: \"c9975a71-d883-4d38-8b69-e3aa21574b13\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" Nov 02 23:35:50.598479 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.598443 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9975a71-d883-4d38-8b69-e3aa21574b13-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89\" (UID: \"c9975a71-d883-4d38-8b69-e3aa21574b13\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" Nov 02 23:35:50.699130 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.699090 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29r8f\" (UniqueName: \"kubernetes.io/projected/00d606e8-8e81-4b50-884b-d4e6bb36659c-kube-api-access-29r8f\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5\" (UID: \"00d606e8-8e81-4b50-884b-d4e6bb36659c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" Nov 02 23:35:50.699130 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.699132 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9975a71-d883-4d38-8b69-e3aa21574b13-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89\" (UID: \"c9975a71-d883-4d38-8b69-e3aa21574b13\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" Nov 02 23:35:50.699701 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.699168 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9975a71-d883-4d38-8b69-e3aa21574b13-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89\" (UID: \"c9975a71-d883-4d38-8b69-e3aa21574b13\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" Nov 02 23:35:50.699701 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.699338 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crmh9\" (UniqueName: \"kubernetes.io/projected/c9975a71-d883-4d38-8b69-e3aa21574b13-kube-api-access-crmh9\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89\" (UID: \"c9975a71-d883-4d38-8b69-e3aa21574b13\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" Nov 02 23:35:50.699701 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.699394 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00d606e8-8e81-4b50-884b-d4e6bb36659c-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5\" (UID: \"00d606e8-8e81-4b50-884b-d4e6bb36659c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" Nov 02 23:35:50.699701 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.699435 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00d606e8-8e81-4b50-884b-d4e6bb36659c-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5\" (UID: \"00d606e8-8e81-4b50-884b-d4e6bb36659c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" Nov 02 23:35:50.699701 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.699460 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9975a71-d883-4d38-8b69-e3aa21574b13-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89\" (UID: \"c9975a71-d883-4d38-8b69-e3aa21574b13\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" Nov 02 23:35:50.699701 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.699581 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9975a71-d883-4d38-8b69-e3aa21574b13-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89\" (UID: \"c9975a71-d883-4d38-8b69-e3aa21574b13\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" Nov 02 23:35:50.699701 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.699625 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9975a71-d883-4d38-8b69-e3aa21574b13-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89\" (UID: \"c9975a71-d883-4d38-8b69-e3aa21574b13\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" Nov 02 23:35:50.699701 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.699685 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9975a71-d883-4d38-8b69-e3aa21574b13-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89\" (UID: \"c9975a71-d883-4d38-8b69-e3aa21574b13\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" Nov 02 23:35:50.700146 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.699629 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9975a71-d883-4d38-8b69-e3aa21574b13-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89\" (UID: \"c9975a71-d883-4d38-8b69-e3aa21574b13\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" Nov 02 23:35:50.700146 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.699722 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00d606e8-8e81-4b50-884b-d4e6bb36659c-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5\" (UID: \"00d606e8-8e81-4b50-884b-d4e6bb36659c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" Nov 02 23:35:50.700146 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.699771 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d606e8-8e81-4b50-884b-d4e6bb36659c-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5\" (UID: \"00d606e8-8e81-4b50-884b-d4e6bb36659c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" Nov 02 23:35:50.700146 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.699811 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9975a71-d883-4d38-8b69-e3aa21574b13-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89\" (UID: \"c9975a71-d883-4d38-8b69-e3aa21574b13\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" Nov 02 23:35:50.700146 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.699849 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9975a71-d883-4d38-8b69-e3aa21574b13-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89\" (UID: \"c9975a71-d883-4d38-8b69-e3aa21574b13\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" Nov 02 23:35:50.700146 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.699846 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00d606e8-8e81-4b50-884b-d4e6bb36659c-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5\" (UID: \"00d606e8-8e81-4b50-884b-d4e6bb36659c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" Nov 02 23:35:50.700146 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.699932 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00d606e8-8e81-4b50-884b-d4e6bb36659c-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5\" (UID: \"00d606e8-8e81-4b50-884b-d4e6bb36659c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" Nov 02 23:35:50.700146 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.699936 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9975a71-d883-4d38-8b69-e3aa21574b13-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89\" (UID: \"c9975a71-d883-4d38-8b69-e3aa21574b13\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" Nov 02 23:35:50.700146 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.699990 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d606e8-8e81-4b50-884b-d4e6bb36659c-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5\" (UID: \"00d606e8-8e81-4b50-884b-d4e6bb36659c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" Nov 02 23:35:50.700146 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.700041 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00d606e8-8e81-4b50-884b-d4e6bb36659c-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5\" (UID: \"00d606e8-8e81-4b50-884b-d4e6bb36659c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" Nov 02 23:35:50.700146 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.700114 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00d606e8-8e81-4b50-884b-d4e6bb36659c-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5\" (UID: \"00d606e8-8e81-4b50-884b-d4e6bb36659c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" Nov 02 23:35:50.700146 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.700120 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d606e8-8e81-4b50-884b-d4e6bb36659c-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5\" (UID: \"00d606e8-8e81-4b50-884b-d4e6bb36659c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" Nov 02 23:35:50.700743 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.700168 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9975a71-d883-4d38-8b69-e3aa21574b13-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89\" (UID: \"c9975a71-d883-4d38-8b69-e3aa21574b13\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" Nov 02 23:35:50.700743 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.700266 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00d606e8-8e81-4b50-884b-d4e6bb36659c-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5\" (UID: \"00d606e8-8e81-4b50-884b-d4e6bb36659c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" Nov 02 23:35:50.700743 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.700302 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9975a71-d883-4d38-8b69-e3aa21574b13-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89\" (UID: \"c9975a71-d883-4d38-8b69-e3aa21574b13\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" Nov 02 23:35:50.700743 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.700535 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d606e8-8e81-4b50-884b-d4e6bb36659c-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5\" (UID: \"00d606e8-8e81-4b50-884b-d4e6bb36659c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" Nov 02 23:35:50.700743 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.700585 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00d606e8-8e81-4b50-884b-d4e6bb36659c-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5\" (UID: \"00d606e8-8e81-4b50-884b-d4e6bb36659c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" Nov 02 23:35:50.701025 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.700803 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9975a71-d883-4d38-8b69-e3aa21574b13-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89\" (UID: \"c9975a71-d883-4d38-8b69-e3aa21574b13\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" Nov 02 23:35:50.702359 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.702332 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00d606e8-8e81-4b50-884b-d4e6bb36659c-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5\" (UID: \"00d606e8-8e81-4b50-884b-d4e6bb36659c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" Nov 02 23:35:50.702508 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.702473 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9975a71-d883-4d38-8b69-e3aa21574b13-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89\" (UID: \"c9975a71-d883-4d38-8b69-e3aa21574b13\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" Nov 02 23:35:50.702747 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.702720 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9975a71-d883-4d38-8b69-e3aa21574b13-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89\" (UID: \"c9975a71-d883-4d38-8b69-e3aa21574b13\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" Nov 02 23:35:50.702873 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.702854 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00d606e8-8e81-4b50-884b-d4e6bb36659c-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5\" (UID: \"00d606e8-8e81-4b50-884b-d4e6bb36659c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" Nov 02 23:35:50.753864 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.753830 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29r8f\" (UniqueName: \"kubernetes.io/projected/00d606e8-8e81-4b50-884b-d4e6bb36659c-kube-api-access-29r8f\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5\" (UID: \"00d606e8-8e81-4b50-884b-d4e6bb36659c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" Nov 02 23:35:50.753864 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.753848 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9975a71-d883-4d38-8b69-e3aa21574b13-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89\" (UID: \"c9975a71-d883-4d38-8b69-e3aa21574b13\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" Nov 02 23:35:50.754077 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.753985 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crmh9\" (UniqueName: \"kubernetes.io/projected/c9975a71-d883-4d38-8b69-e3aa21574b13-kube-api-access-crmh9\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89\" (UID: \"c9975a71-d883-4d38-8b69-e3aa21574b13\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" Nov 02 23:35:50.755687 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.755670 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" Nov 02 23:35:50.800131 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:50.800098 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00d606e8-8e81-4b50-884b-d4e6bb36659c-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5\" (UID: \"00d606e8-8e81-4b50-884b-d4e6bb36659c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" Nov 02 23:35:51.031059 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:51.031024 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" Nov 02 23:35:51.149649 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:51.149619 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8d06dd9-47b5-442a-ad96-f8616e6570a8" path="/var/lib/kubelet/pods/e8d06dd9-47b5-442a-ad96-f8616e6570a8/volumes" Nov 02 23:35:51.171423 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:51.171396 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89"] Nov 02 23:35:51.172180 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:35:51.172151 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9975a71_d883_4d38_8b69_e3aa21574b13.slice/crio-001944fb8e4a8891bf2c12ae3911c0e3f30de18eab85e0511c2b00c7888a6b44 WatchSource:0}: Error finding container 001944fb8e4a8891bf2c12ae3911c0e3f30de18eab85e0511c2b00c7888a6b44: Status 404 returned error can't find the container with id 001944fb8e4a8891bf2c12ae3911c0e3f30de18eab85e0511c2b00c7888a6b44 Nov 02 23:35:51.174630 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:51.174600 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:35:51.174737 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:51.174663 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:35:51.174737 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:51.174690 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:35:51.334008 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:51.333877 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" event={"ID":"c9975a71-d883-4d38-8b69-e3aa21574b13","Type":"ContainerStarted","Data":"4be028871e0b895ca7dcb8e38e7ca595eec460160446c8915ba0fb4b8b61ff74"} Nov 02 23:35:51.334008 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:51.333964 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" event={"ID":"c9975a71-d883-4d38-8b69-e3aa21574b13","Type":"ContainerStarted","Data":"001944fb8e4a8891bf2c12ae3911c0e3f30de18eab85e0511c2b00c7888a6b44"} Nov 02 23:35:51.402523 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:51.402488 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5"] Nov 02 23:35:51.404050 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:35:51.404025 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00d606e8_8e81_4b50_884b_d4e6bb36659c.slice/crio-124c0964ad0848e70df348f32f6a2a2e48250565944692525dbfd362e8dfe6a9 WatchSource:0}: Error finding container 124c0964ad0848e70df348f32f6a2a2e48250565944692525dbfd362e8dfe6a9: Status 404 returned error can't find the container with id 124c0964ad0848e70df348f32f6a2a2e48250565944692525dbfd362e8dfe6a9 Nov 02 23:35:51.407717 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:51.407675 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:35:51.407797 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:51.407761 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:35:51.407854 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:51.407828 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:35:51.446041 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:51.445995 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" podStartSLOduration=1.445977323 podStartE2EDuration="1.445977323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:35:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:35:51.44404039 +0000 UTC m=+2600.412572198" watchObservedRunningTime="2026-11-02 23:35:51.445977323 +0000 UTC m=+2600.414509108" Nov 02 23:35:51.519602 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:51.519567 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm"] Nov 02 23:35:51.519844 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:51.519819 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm" podUID="145c9784-8db4-4686-b77d-7f33d51272d0" containerName="maas-api" containerID="cri-o://258ca72e34619731e50617d34660013bad571136e81dbb8147bf5b775afb6bda" gracePeriod=30 Nov 02 23:35:51.776776 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:51.756837 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" Nov 02 23:35:51.850476 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:51.850444 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm" Nov 02 23:35:51.933960 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:51.933768 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/145c9784-8db4-4686-b77d-7f33d51272d0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/145c9784-8db4-4686-b77d-7f33d51272d0-maas-api-tls\") pod \"145c9784-8db4-4686-b77d-7f33d51272d0\" (UID: \"145c9784-8db4-4686-b77d-7f33d51272d0\") " Nov 02 23:35:51.933960 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:51.933867 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/145c9784-8db4-4686-b77d-7f33d51272d0-kube-api-access-hwqfx\" (UniqueName: \"kubernetes.io/projected/145c9784-8db4-4686-b77d-7f33d51272d0-kube-api-access-hwqfx\") pod \"145c9784-8db4-4686-b77d-7f33d51272d0\" (UID: \"145c9784-8db4-4686-b77d-7f33d51272d0\") " Nov 02 23:35:51.934196 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:51.934101 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/145c9784-8db4-4686-b77d-7f33d51272d0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/145c9784-8db4-4686-b77d-7f33d51272d0-openshift-service-ca\") pod \"145c9784-8db4-4686-b77d-7f33d51272d0\" (UID: \"145c9784-8db4-4686-b77d-7f33d51272d0\") " Nov 02 23:35:51.934196 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:51.934145 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/145c9784-8db4-4686-b77d-7f33d51272d0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/145c9784-8db4-4686-b77d-7f33d51272d0-metrics-certs\") pod \"145c9784-8db4-4686-b77d-7f33d51272d0\" (UID: \"145c9784-8db4-4686-b77d-7f33d51272d0\") " Nov 02 23:35:51.944533 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:51.938294 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/145c9784-8db4-4686-b77d-7f33d51272d0-openshift-service-ca" pod "145c9784-8db4-4686-b77d-7f33d51272d0" (UID: "145c9784-8db4-4686-b77d-7f33d51272d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:35:51.955954 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:51.955137 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/145c9784-8db4-4686-b77d-7f33d51272d0-kube-api-access-hwqfx" pod "145c9784-8db4-4686-b77d-7f33d51272d0" (UID: "145c9784-8db4-4686-b77d-7f33d51272d0"). InnerVolumeSpecName "kube-api-access-hwqfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:35:51.962769 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:51.961064 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/145c9784-8db4-4686-b77d-7f33d51272d0-maas-api-tls" pod "145c9784-8db4-4686-b77d-7f33d51272d0" (UID: "145c9784-8db4-4686-b77d-7f33d51272d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:35:51.980154 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:51.975517 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/145c9784-8db4-4686-b77d-7f33d51272d0-metrics-certs" pod "145c9784-8db4-4686-b77d-7f33d51272d0" (UID: "145c9784-8db4-4686-b77d-7f33d51272d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:35:52.035560 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:52.035517 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/145c9784-8db4-4686-b77d-7f33d51272d0-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:52.035560 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:52.035556 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/145c9784-8db4-4686-b77d-7f33d51272d0-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:52.035560 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:52.035567 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/145c9784-8db4-4686-b77d-7f33d51272d0-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:52.035800 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:52.035578 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwqfx\" (UniqueName: \"kubernetes.io/projected/145c9784-8db4-4686-b77d-7f33d51272d0-kube-api-access-hwqfx\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:52.393971 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:52.392462 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" event={"ID":"00d606e8-8e81-4b50-884b-d4e6bb36659c","Type":"ContainerStarted","Data":"813a3b75bfe8737b09a3cbba8a0c3f19069c4b5c4dc6c0f19aab2c73237b6011"} Nov 02 23:35:52.393971 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:52.392531 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" event={"ID":"00d606e8-8e81-4b50-884b-d4e6bb36659c","Type":"ContainerStarted","Data":"124c0964ad0848e70df348f32f6a2a2e48250565944692525dbfd362e8dfe6a9"} Nov 02 23:35:52.398051 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:52.395530 5076 generic.go:356] "Generic (PLEG): container finished" podID="145c9784-8db4-4686-b77d-7f33d51272d0" containerID="258ca72e34619731e50617d34660013bad571136e81dbb8147bf5b775afb6bda" exitCode=0 Nov 02 23:35:52.398051 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:52.396460 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm" Nov 02 23:35:52.398051 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:52.396845 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm" event={"ID":"145c9784-8db4-4686-b77d-7f33d51272d0","Type":"ContainerDied","Data":"258ca72e34619731e50617d34660013bad571136e81dbb8147bf5b775afb6bda"} Nov 02 23:35:52.398051 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:52.397142 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm" event={"ID":"145c9784-8db4-4686-b77d-7f33d51272d0","Type":"ContainerDied","Data":"a960e7f0235073f1af02e0ac7256e13b4ef7853f2d9a8ced941c30c72f753384"} Nov 02 23:35:52.398051 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:52.397209 5076 scope.go:122] "RemoveContainer" containerID="258ca72e34619731e50617d34660013bad571136e81dbb8147bf5b775afb6bda" Nov 02 23:35:52.431471 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:52.431152 5076 scope.go:122] "RemoveContainer" containerID="258ca72e34619731e50617d34660013bad571136e81dbb8147bf5b775afb6bda" Nov 02 23:35:52.431857 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:35:52.431763 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"258ca72e34619731e50617d34660013bad571136e81dbb8147bf5b775afb6bda\": container with ID starting with 258ca72e34619731e50617d34660013bad571136e81dbb8147bf5b775afb6bda not found: ID does not exist" containerID="258ca72e34619731e50617d34660013bad571136e81dbb8147bf5b775afb6bda" Nov 02 23:35:52.431857 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:52.431806 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"258ca72e34619731e50617d34660013bad571136e81dbb8147bf5b775afb6bda"} err="failed to get container status \"258ca72e34619731e50617d34660013bad571136e81dbb8147bf5b775afb6bda\": rpc error: code = NotFound desc = could not find container \"258ca72e34619731e50617d34660013bad571136e81dbb8147bf5b775afb6bda\": container with ID starting with 258ca72e34619731e50617d34660013bad571136e81dbb8147bf5b775afb6bda not found: ID does not exist" Nov 02 23:35:52.547916 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:52.547299 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" podStartSLOduration=2.547279384 podStartE2EDuration="2.547279384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:35:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:35:52.544901857 +0000 UTC m=+2601.513433636" watchObservedRunningTime="2026-11-02 23:35:52.547279384 +0000 UTC m=+2601.515811169" Nov 02 23:35:52.678091 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:52.677981 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm"] Nov 02 23:35:52.681565 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:52.681536 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-81c3c9-6fcbc46b58-fkjhm"] Nov 02 23:35:52.761382 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:52.761280 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 02 23:35:52.761382 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:52.761366 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" podUID="c9975a71-d883-4d38-8b69-e3aa21574b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 02 23:35:53.036915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:53.036820 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" Nov 02 23:35:53.037435 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:53.036969 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 23:35:53.037435 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:53.037012 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" podUID="00d606e8-8e81-4b50-884b-d4e6bb36659c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 23:35:53.165077 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:53.165041 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="145c9784-8db4-4686-b77d-7f33d51272d0" path="/var/lib/kubelet/pods/145c9784-8db4-4686-b77d-7f33d51272d0/volumes" Nov 02 23:35:53.757916 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:53.756352 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 02 23:35:53.757916 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:53.756416 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" podUID="c9975a71-d883-4d38-8b69-e3aa21574b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 02 23:35:54.037968 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:54.032311 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 23:35:54.037968 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:54.032375 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" podUID="00d606e8-8e81-4b50-884b-d4e6bb36659c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 23:35:54.194592 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:54.194553 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m"] Nov 02 23:35:54.194819 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:54.194794 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" podUID="8bcab402-b870-4284-9d80-a7c8243492bd" containerName="istio-proxy" containerID="cri-o://f742ef537c7215ea5da6f9eb2fd32e095b6cbc434e2704b06ef5bfea32dbf1c5" gracePeriod=30 Nov 02 23:35:54.756365 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:54.756324 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 02 23:35:54.756716 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:54.756682 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" podUID="c9975a71-d883-4d38-8b69-e3aa21574b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 02 23:35:55.037970 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:55.037410 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 23:35:55.037970 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:55.037475 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" podUID="00d606e8-8e81-4b50-884b-d4e6bb36659c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 23:35:55.761915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:55.756227 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 02 23:35:55.761915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:55.756302 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" podUID="c9975a71-d883-4d38-8b69-e3aa21574b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 02 23:35:56.041974 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:56.033319 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 23:35:56.041974 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:56.033388 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" podUID="00d606e8-8e81-4b50-884b-d4e6bb36659c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 23:35:56.764845 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:56.764346 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 02 23:35:56.764845 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:56.764415 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" podUID="c9975a71-d883-4d38-8b69-e3aa21574b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 02 23:35:57.034807 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:57.034712 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 23:35:57.035123 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:57.035097 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" podUID="00d606e8-8e81-4b50-884b-d4e6bb36659c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 23:35:57.303485 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:57.303404 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-b9756748c-pqkdd" Nov 02 23:35:57.756485 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:57.756448 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 02 23:35:57.756676 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:57.756507 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" podUID="c9975a71-d883-4d38-8b69-e3aa21574b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 02 23:35:58.032502 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:58.032410 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 23:35:58.032502 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:58.032481 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" podUID="00d606e8-8e81-4b50-884b-d4e6bb36659c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 23:35:58.308478 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:58.308398 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5d87-598f6cbf4d-2m2xw" Nov 02 23:35:59.065239 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.065206 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" Nov 02 23:35:59.065960 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.065933 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" Nov 02 23:35:59.066794 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.066773 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" Nov 02 23:35:59.259546 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.259504 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" Nov 02 23:35:59.259943 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.259922 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" Nov 02 23:35:59.260781 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.260756 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" Nov 02 23:35:59.487693 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.487662 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" Nov 02 23:35:59.491913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.491865 5076 generic.go:356] "Generic (PLEG): container finished" podID="8bcab402-b870-4284-9d80-a7c8243492bd" containerID="f742ef537c7215ea5da6f9eb2fd32e095b6cbc434e2704b06ef5bfea32dbf1c5" exitCode=0 Nov 02 23:35:59.492074 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.491937 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" event={"ID":"8bcab402-b870-4284-9d80-a7c8243492bd","Type":"ContainerDied","Data":"f742ef537c7215ea5da6f9eb2fd32e095b6cbc434e2704b06ef5bfea32dbf1c5"} Nov 02 23:35:59.492074 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.491992 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" event={"ID":"8bcab402-b870-4284-9d80-a7c8243492bd","Type":"ContainerDied","Data":"46456ba57e22fda260e794ae84fac28e0c995ef5c3e274d2b7a6e8efc12453c1"} Nov 02 23:35:59.492074 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.492016 5076 scope.go:122] "RemoveContainer" containerID="f742ef537c7215ea5da6f9eb2fd32e095b6cbc434e2704b06ef5bfea32dbf1c5" Nov 02 23:35:59.492074 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.491943 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m" Nov 02 23:35:59.502661 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.502638 5076 scope.go:122] "RemoveContainer" containerID="8c6b87f5d8752b4cfb609abe74bb9468ddeaedd3545e3d93c816bf11b3814cc4" Nov 02 23:35:59.511664 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.511644 5076 scope.go:122] "RemoveContainer" containerID="f742ef537c7215ea5da6f9eb2fd32e095b6cbc434e2704b06ef5bfea32dbf1c5" Nov 02 23:35:59.511956 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:35:59.511935 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f742ef537c7215ea5da6f9eb2fd32e095b6cbc434e2704b06ef5bfea32dbf1c5\": container with ID starting with f742ef537c7215ea5da6f9eb2fd32e095b6cbc434e2704b06ef5bfea32dbf1c5 not found: ID does not exist" containerID="f742ef537c7215ea5da6f9eb2fd32e095b6cbc434e2704b06ef5bfea32dbf1c5" Nov 02 23:35:59.512043 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.511971 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f742ef537c7215ea5da6f9eb2fd32e095b6cbc434e2704b06ef5bfea32dbf1c5"} err="failed to get container status \"f742ef537c7215ea5da6f9eb2fd32e095b6cbc434e2704b06ef5bfea32dbf1c5\": rpc error: code = NotFound desc = could not find container \"f742ef537c7215ea5da6f9eb2fd32e095b6cbc434e2704b06ef5bfea32dbf1c5\": container with ID starting with f742ef537c7215ea5da6f9eb2fd32e095b6cbc434e2704b06ef5bfea32dbf1c5 not found: ID does not exist" Nov 02 23:35:59.512043 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.511995 5076 scope.go:122] "RemoveContainer" containerID="8c6b87f5d8752b4cfb609abe74bb9468ddeaedd3545e3d93c816bf11b3814cc4" Nov 02 23:35:59.512241 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:35:59.512224 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c6b87f5d8752b4cfb609abe74bb9468ddeaedd3545e3d93c816bf11b3814cc4\": container with ID starting with 8c6b87f5d8752b4cfb609abe74bb9468ddeaedd3545e3d93c816bf11b3814cc4 not found: ID does not exist" containerID="8c6b87f5d8752b4cfb609abe74bb9468ddeaedd3545e3d93c816bf11b3814cc4" Nov 02 23:35:59.512298 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.512249 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c6b87f5d8752b4cfb609abe74bb9468ddeaedd3545e3d93c816bf11b3814cc4"} err="failed to get container status \"8c6b87f5d8752b4cfb609abe74bb9468ddeaedd3545e3d93c816bf11b3814cc4\": rpc error: code = NotFound desc = could not find container \"8c6b87f5d8752b4cfb609abe74bb9468ddeaedd3545e3d93c816bf11b3814cc4\": container with ID starting with 8c6b87f5d8752b4cfb609abe74bb9468ddeaedd3545e3d93c816bf11b3814cc4 not found: ID does not exist" Nov 02 23:35:59.522333 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.522304 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bcab402-b870-4284-9d80-a7c8243492bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bcab402-b870-4284-9d80-a7c8243492bd-workload-certs\") pod \"8bcab402-b870-4284-9d80-a7c8243492bd\" (UID: \"8bcab402-b870-4284-9d80-a7c8243492bd\") " Nov 02 23:35:59.522507 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.522402 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bcab402-b870-4284-9d80-a7c8243492bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bcab402-b870-4284-9d80-a7c8243492bd-istiod-ca-cert\") pod \"8bcab402-b870-4284-9d80-a7c8243492bd\" (UID: \"8bcab402-b870-4284-9d80-a7c8243492bd\") " Nov 02 23:35:59.522507 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.522437 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bcab402-b870-4284-9d80-a7c8243492bd-kube-api-access-bgmzx\" (UniqueName: \"kubernetes.io/projected/8bcab402-b870-4284-9d80-a7c8243492bd-kube-api-access-bgmzx\") pod \"8bcab402-b870-4284-9d80-a7c8243492bd\" (UID: \"8bcab402-b870-4284-9d80-a7c8243492bd\") " Nov 02 23:35:59.522507 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.522499 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bcab402-b870-4284-9d80-a7c8243492bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bcab402-b870-4284-9d80-a7c8243492bd-credential-socket\") pod \"8bcab402-b870-4284-9d80-a7c8243492bd\" (UID: \"8bcab402-b870-4284-9d80-a7c8243492bd\") " Nov 02 23:35:59.522693 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.522522 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bcab402-b870-4284-9d80-a7c8243492bd-workload-certs" pod "8bcab402-b870-4284-9d80-a7c8243492bd" (UID: "8bcab402-b870-4284-9d80-a7c8243492bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:35:59.522693 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.522536 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8bcab402-b870-4284-9d80-a7c8243492bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bcab402-b870-4284-9d80-a7c8243492bd-istio-podinfo\") pod \"8bcab402-b870-4284-9d80-a7c8243492bd\" (UID: \"8bcab402-b870-4284-9d80-a7c8243492bd\") " Nov 02 23:35:59.522693 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.522667 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bcab402-b870-4284-9d80-a7c8243492bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bcab402-b870-4284-9d80-a7c8243492bd-workload-socket\") pod \"8bcab402-b870-4284-9d80-a7c8243492bd\" (UID: \"8bcab402-b870-4284-9d80-a7c8243492bd\") " Nov 02 23:35:59.522855 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.522693 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bcab402-b870-4284-9d80-a7c8243492bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bcab402-b870-4284-9d80-a7c8243492bd-istio-data\") pod \"8bcab402-b870-4284-9d80-a7c8243492bd\" (UID: \"8bcab402-b870-4284-9d80-a7c8243492bd\") " Nov 02 23:35:59.522855 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.522735 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bcab402-b870-4284-9d80-a7c8243492bd-istio-token\" (UniqueName: \"kubernetes.io/projected/8bcab402-b870-4284-9d80-a7c8243492bd-istio-token\") pod \"8bcab402-b870-4284-9d80-a7c8243492bd\" (UID: \"8bcab402-b870-4284-9d80-a7c8243492bd\") " Nov 02 23:35:59.522855 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.522828 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bcab402-b870-4284-9d80-a7c8243492bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bcab402-b870-4284-9d80-a7c8243492bd-istio-envoy\") pod \"8bcab402-b870-4284-9d80-a7c8243492bd\" (UID: \"8bcab402-b870-4284-9d80-a7c8243492bd\") " Nov 02 23:35:59.523142 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.523119 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bcab402-b870-4284-9d80-a7c8243492bd-istiod-ca-cert" pod "8bcab402-b870-4284-9d80-a7c8243492bd" (UID: "8bcab402-b870-4284-9d80-a7c8243492bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:35:59.523142 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.523140 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bcab402-b870-4284-9d80-a7c8243492bd-workload-socket" pod "8bcab402-b870-4284-9d80-a7c8243492bd" (UID: "8bcab402-b870-4284-9d80-a7c8243492bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:35:59.523340 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.523307 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bcab402-b870-4284-9d80-a7c8243492bd-istio-data" pod "8bcab402-b870-4284-9d80-a7c8243492bd" (UID: "8bcab402-b870-4284-9d80-a7c8243492bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:35:59.524027 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.524001 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bcab402-b870-4284-9d80-a7c8243492bd-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:59.525016 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.524994 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bcab402-b870-4284-9d80-a7c8243492bd-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:59.525120 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.525018 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bcab402-b870-4284-9d80-a7c8243492bd-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:59.525120 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.525034 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bcab402-b870-4284-9d80-a7c8243492bd-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:59.525120 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.524950 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bcab402-b870-4284-9d80-a7c8243492bd-credential-socket" pod "8bcab402-b870-4284-9d80-a7c8243492bd" (UID: "8bcab402-b870-4284-9d80-a7c8243492bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:35:59.525672 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.525645 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8bcab402-b870-4284-9d80-a7c8243492bd-istio-podinfo" pod "8bcab402-b870-4284-9d80-a7c8243492bd" (UID: "8bcab402-b870-4284-9d80-a7c8243492bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:35:59.526097 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.526078 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bcab402-b870-4284-9d80-a7c8243492bd-istio-envoy" pod "8bcab402-b870-4284-9d80-a7c8243492bd" (UID: "8bcab402-b870-4284-9d80-a7c8243492bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:35:59.526170 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.526133 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bcab402-b870-4284-9d80-a7c8243492bd-kube-api-access-bgmzx" pod "8bcab402-b870-4284-9d80-a7c8243492bd" (UID: "8bcab402-b870-4284-9d80-a7c8243492bd"). InnerVolumeSpecName "kube-api-access-bgmzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:35:59.526241 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.526226 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bcab402-b870-4284-9d80-a7c8243492bd-istio-token" pod "8bcab402-b870-4284-9d80-a7c8243492bd" (UID: "8bcab402-b870-4284-9d80-a7c8243492bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:35:59.626094 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.626017 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgmzx\" (UniqueName: \"kubernetes.io/projected/8bcab402-b870-4284-9d80-a7c8243492bd-kube-api-access-bgmzx\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:59.626094 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.626047 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bcab402-b870-4284-9d80-a7c8243492bd-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:59.626094 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.626056 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bcab402-b870-4284-9d80-a7c8243492bd-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:59.626094 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.626065 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bcab402-b870-4284-9d80-a7c8243492bd-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:59.626094 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.626076 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bcab402-b870-4284-9d80-a7c8243492bd-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:59.908151 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.908056 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m"] Nov 02 23:35:59.912662 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:35:59.912632 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-81c3c9-openshift-default-6c94744c57-q7n8m"] Nov 02 23:36:01.149126 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:01.149092 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bcab402-b870-4284-9d80-a7c8243492bd" path="/var/lib/kubelet/pods/8bcab402-b870-4284-9d80-a7c8243492bd/volumes" Nov 02 23:36:09.884019 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:36:09.883985 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:36:35.081778 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:35.081693 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76"] Nov 02 23:36:35.082292 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:35.081999 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76" podUID="3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766" containerName="maas-api" containerID="cri-o://a9c5024d89b7843c06e82166e90ce77716026d40f1701386e8d05eb1340075df" gracePeriod=30 Nov 02 23:36:35.385285 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:35.385257 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76" Nov 02 23:36:35.521633 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:35.521590 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766-maas-api-tls\") pod \"3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766\" (UID: \"3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766\") " Nov 02 23:36:35.521806 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:35.521687 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766-metrics-certs\") pod \"3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766\" (UID: \"3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766\") " Nov 02 23:36:35.521806 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:35.521742 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766-openshift-service-ca\") pod \"3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766\" (UID: \"3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766\") " Nov 02 23:36:35.521806 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:35.521775 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766-kube-api-access-b2f4p\" (UniqueName: \"kubernetes.io/projected/3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766-kube-api-access-b2f4p\") pod \"3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766\" (UID: \"3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766\") " Nov 02 23:36:35.522164 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:35.522140 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766-openshift-service-ca" pod "3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766" (UID: "3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:36:35.522365 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:35.522349 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:35.524377 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:35.524340 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766-kube-api-access-b2f4p" pod "3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766" (UID: "3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766"). InnerVolumeSpecName "kube-api-access-b2f4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:36:35.525393 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:35.525369 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766-maas-api-tls" pod "3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766" (UID: "3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:36:35.525591 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:35.525574 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766-metrics-certs" pod "3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766" (UID: "3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:36:35.558185 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:35.558154 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db5d87-598f6cbf4d-2m2xw"] Nov 02 23:36:35.558418 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:35.558390 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5d87-598f6cbf4d-2m2xw" podUID="e02e672a-a973-4706-9c2b-a307aabafa9b" containerName="payload-processing" containerID="cri-o://4ce532c18fa6066c4db7ad370392691aade147378c650c741472922e7ebc864b" gracePeriod=30 Nov 02 23:36:35.613566 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:35.613501 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-b9756748c-pqkdd"] Nov 02 23:36:35.613704 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:35.613684 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-b9756748c-pqkdd" podUID="8895d1e5-9514-4125-8343-e8d3c9cf3d09" containerName="payload-pre-processing" containerID="cri-o://61db32bc96293dafdb77b3503283337e84c28f3f636fc48fa2450caf135b5962" gracePeriod=30 Nov 02 23:36:35.623742 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:35.623717 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:35.623742 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:35.623742 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:35.623880 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:35.623753 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2f4p\" (UniqueName: \"kubernetes.io/projected/3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766-kube-api-access-b2f4p\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:35.738121 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:35.736539 5076 generic.go:356] "Generic (PLEG): container finished" podID="e02e672a-a973-4706-9c2b-a307aabafa9b" containerID="4ce532c18fa6066c4db7ad370392691aade147378c650c741472922e7ebc864b" exitCode=0 Nov 02 23:36:35.738121 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:35.736689 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5d87-598f6cbf4d-2m2xw" event={"ID":"e02e672a-a973-4706-9c2b-a307aabafa9b","Type":"ContainerDied","Data":"4ce532c18fa6066c4db7ad370392691aade147378c650c741472922e7ebc864b"} Nov 02 23:36:35.739139 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:35.738477 5076 generic.go:356] "Generic (PLEG): container finished" podID="3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766" containerID="a9c5024d89b7843c06e82166e90ce77716026d40f1701386e8d05eb1340075df" exitCode=0 Nov 02 23:36:35.739139 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:35.738544 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76" Nov 02 23:36:35.739139 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:35.738599 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76" event={"ID":"3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766","Type":"ContainerDied","Data":"a9c5024d89b7843c06e82166e90ce77716026d40f1701386e8d05eb1340075df"} Nov 02 23:36:35.739139 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:35.738658 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76" event={"ID":"3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766","Type":"ContainerDied","Data":"1fb6278f27046a7d3ead0e295899512fabc8c3f7c6b010c657fd8750d3acef0e"} Nov 02 23:36:35.739139 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:35.738690 5076 scope.go:122] "RemoveContainer" containerID="a9c5024d89b7843c06e82166e90ce77716026d40f1701386e8d05eb1340075df" Nov 02 23:36:35.754241 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:35.754116 5076 scope.go:122] "RemoveContainer" containerID="a9c5024d89b7843c06e82166e90ce77716026d40f1701386e8d05eb1340075df" Nov 02 23:36:35.754544 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:36:35.754452 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9c5024d89b7843c06e82166e90ce77716026d40f1701386e8d05eb1340075df\": container with ID starting with a9c5024d89b7843c06e82166e90ce77716026d40f1701386e8d05eb1340075df not found: ID does not exist" containerID="a9c5024d89b7843c06e82166e90ce77716026d40f1701386e8d05eb1340075df" Nov 02 23:36:35.754544 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:35.754488 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9c5024d89b7843c06e82166e90ce77716026d40f1701386e8d05eb1340075df"} err="failed to get container status \"a9c5024d89b7843c06e82166e90ce77716026d40f1701386e8d05eb1340075df\": rpc error: code = NotFound desc = could not find container \"a9c5024d89b7843c06e82166e90ce77716026d40f1701386e8d05eb1340075df\": container with ID starting with a9c5024d89b7843c06e82166e90ce77716026d40f1701386e8d05eb1340075df not found: ID does not exist" Nov 02 23:36:35.879858 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:35.879821 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5d87-598f6cbf4d-2m2xw" Nov 02 23:36:35.896145 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:35.895377 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76"] Nov 02 23:36:35.899622 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:35.897875 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76"] Nov 02 23:36:35.911125 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:35.911100 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-b9756748c-pqkdd" Nov 02 23:36:36.035653 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:36.035624 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e02e672a-a973-4706-9c2b-a307aabafa9b-kube-api-access-jjdzh\" (UniqueName: \"kubernetes.io/projected/e02e672a-a973-4706-9c2b-a307aabafa9b-kube-api-access-jjdzh\") pod \"e02e672a-a973-4706-9c2b-a307aabafa9b\" (UID: \"e02e672a-a973-4706-9c2b-a307aabafa9b\") " Nov 02 23:36:36.035848 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:36.035683 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e02e672a-a973-4706-9c2b-a307aabafa9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e02e672a-a973-4706-9c2b-a307aabafa9b-plugins-config-volume\") pod \"e02e672a-a973-4706-9c2b-a307aabafa9b\" (UID: \"e02e672a-a973-4706-9c2b-a307aabafa9b\") " Nov 02 23:36:36.035848 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:36.035716 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8895d1e5-9514-4125-8343-e8d3c9cf3d09-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8895d1e5-9514-4125-8343-e8d3c9cf3d09-plugins-config-volume\") pod \"8895d1e5-9514-4125-8343-e8d3c9cf3d09\" (UID: \"8895d1e5-9514-4125-8343-e8d3c9cf3d09\") " Nov 02 23:36:36.035848 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:36.035745 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8895d1e5-9514-4125-8343-e8d3c9cf3d09-kube-api-access-mm68l\" (UniqueName: \"kubernetes.io/projected/8895d1e5-9514-4125-8343-e8d3c9cf3d09-kube-api-access-mm68l\") pod \"8895d1e5-9514-4125-8343-e8d3c9cf3d09\" (UID: \"8895d1e5-9514-4125-8343-e8d3c9cf3d09\") " Nov 02 23:36:36.036181 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:36.036155 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8895d1e5-9514-4125-8343-e8d3c9cf3d09-plugins-config-volume" pod "8895d1e5-9514-4125-8343-e8d3c9cf3d09" (UID: "8895d1e5-9514-4125-8343-e8d3c9cf3d09"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:36:36.036304 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:36.036177 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e02e672a-a973-4706-9c2b-a307aabafa9b-plugins-config-volume" pod "e02e672a-a973-4706-9c2b-a307aabafa9b" (UID: "e02e672a-a973-4706-9c2b-a307aabafa9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:36:36.036304 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:36.036275 5076 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e02e672a-a973-4706-9c2b-a307aabafa9b-plugins-config-volume\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:36.036304 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:36.036297 5076 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8895d1e5-9514-4125-8343-e8d3c9cf3d09-plugins-config-volume\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:36.037898 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:36.037869 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e02e672a-a973-4706-9c2b-a307aabafa9b-kube-api-access-jjdzh" pod "e02e672a-a973-4706-9c2b-a307aabafa9b" (UID: "e02e672a-a973-4706-9c2b-a307aabafa9b"). InnerVolumeSpecName "kube-api-access-jjdzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:36:36.037974 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:36.037881 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8895d1e5-9514-4125-8343-e8d3c9cf3d09-kube-api-access-mm68l" pod "8895d1e5-9514-4125-8343-e8d3c9cf3d09" (UID: "8895d1e5-9514-4125-8343-e8d3c9cf3d09"). InnerVolumeSpecName "kube-api-access-mm68l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:36:36.149301 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:36.147335 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjdzh\" (UniqueName: \"kubernetes.io/projected/e02e672a-a973-4706-9c2b-a307aabafa9b-kube-api-access-jjdzh\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:36.149301 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:36.147373 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm68l\" (UniqueName: \"kubernetes.io/projected/8895d1e5-9514-4125-8343-e8d3c9cf3d09-kube-api-access-mm68l\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:36.745710 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:36.745668 5076 generic.go:356] "Generic (PLEG): container finished" podID="8895d1e5-9514-4125-8343-e8d3c9cf3d09" containerID="61db32bc96293dafdb77b3503283337e84c28f3f636fc48fa2450caf135b5962" exitCode=0 Nov 02 23:36:36.745938 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:36.745738 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-b9756748c-pqkdd" Nov 02 23:36:36.745938 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:36.745757 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-b9756748c-pqkdd" event={"ID":"8895d1e5-9514-4125-8343-e8d3c9cf3d09","Type":"ContainerDied","Data":"61db32bc96293dafdb77b3503283337e84c28f3f636fc48fa2450caf135b5962"} Nov 02 23:36:36.745938 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:36.745847 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-b9756748c-pqkdd" event={"ID":"8895d1e5-9514-4125-8343-e8d3c9cf3d09","Type":"ContainerDied","Data":"0873c2e81bb6b2ea7f54e16d8967c1c6436417e24edecf86373eb7faeca50037"} Nov 02 23:36:36.745938 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:36.745874 5076 scope.go:122] "RemoveContainer" containerID="61db32bc96293dafdb77b3503283337e84c28f3f636fc48fa2450caf135b5962" Nov 02 23:36:36.747322 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:36.747185 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5d87-598f6cbf4d-2m2xw" event={"ID":"e02e672a-a973-4706-9c2b-a307aabafa9b","Type":"ContainerDied","Data":"e5fae53e3afd6c92f045f94679138ecd3d4b147df3fe6be990b60a4663a9fc26"} Nov 02 23:36:36.747322 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:36.747194 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5d87-598f6cbf4d-2m2xw" Nov 02 23:36:36.756312 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:36.756295 5076 scope.go:122] "RemoveContainer" containerID="61db32bc96293dafdb77b3503283337e84c28f3f636fc48fa2450caf135b5962" Nov 02 23:36:36.756767 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:36:36.756542 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61db32bc96293dafdb77b3503283337e84c28f3f636fc48fa2450caf135b5962\": container with ID starting with 61db32bc96293dafdb77b3503283337e84c28f3f636fc48fa2450caf135b5962 not found: ID does not exist" containerID="61db32bc96293dafdb77b3503283337e84c28f3f636fc48fa2450caf135b5962" Nov 02 23:36:36.756767 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:36.756571 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61db32bc96293dafdb77b3503283337e84c28f3f636fc48fa2450caf135b5962"} err="failed to get container status \"61db32bc96293dafdb77b3503283337e84c28f3f636fc48fa2450caf135b5962\": rpc error: code = NotFound desc = could not find container \"61db32bc96293dafdb77b3503283337e84c28f3f636fc48fa2450caf135b5962\": container with ID starting with 61db32bc96293dafdb77b3503283337e84c28f3f636fc48fa2450caf135b5962 not found: ID does not exist" Nov 02 23:36:36.756767 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:36.756586 5076 scope.go:122] "RemoveContainer" containerID="4ce532c18fa6066c4db7ad370392691aade147378c650c741472922e7ebc864b" Nov 02 23:36:36.779148 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:36.779124 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-981724-d5985fb4-s88v5"] Nov 02 23:36:36.779333 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:36.779312 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-981724-d5985fb4-s88v5" podUID="d5a2fe82-afb7-4822-b714-d44da311b6ed" containerName="payload-processing" containerID="cri-o://501f4b1a7943ed29522d56ae1682af3b4b9c3e13938461db3e682e82ad03c260" gracePeriod=30 Nov 02 23:36:36.833815 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:36.833783 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5bf989ccc5-mgxq9"] Nov 02 23:36:36.834035 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:36.833993 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5bf989ccc5-mgxq9" podUID="fb5eb5ca-ea96-4366-b376-3abac4ce1d8a" containerName="payload-pre-processing" containerID="cri-o://263d8d00024e4886c14e6a4560721270211610b4fedb6619fdde5209378a6056" gracePeriod=30 Nov 02 23:36:36.904987 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:36.904948 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db5d87-598f6cbf4d-2m2xw"] Nov 02 23:36:36.907577 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:36.907550 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db5d87-598f6cbf4d-2m2xw"] Nov 02 23:36:37.009773 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:37.009728 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-b9756748c-pqkdd"] Nov 02 23:36:37.018954 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:37.017192 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-b9756748c-pqkdd"] Nov 02 23:36:37.142573 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:37.142550 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-981724-d5985fb4-s88v5" Nov 02 23:36:37.145697 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:37.145676 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5bf989ccc5-mgxq9" Nov 02 23:36:37.148195 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:37.148172 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766" path="/var/lib/kubelet/pods/3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766/volumes" Nov 02 23:36:37.148562 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:37.148548 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8895d1e5-9514-4125-8343-e8d3c9cf3d09" path="/var/lib/kubelet/pods/8895d1e5-9514-4125-8343-e8d3c9cf3d09/volumes" Nov 02 23:36:37.149050 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:37.149031 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e02e672a-a973-4706-9c2b-a307aabafa9b" path="/var/lib/kubelet/pods/e02e672a-a973-4706-9c2b-a307aabafa9b/volumes" Nov 02 23:36:37.257575 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:37.257537 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5a2fe82-afb7-4822-b714-d44da311b6ed-kube-api-access-vhv9v\" (UniqueName: \"kubernetes.io/projected/d5a2fe82-afb7-4822-b714-d44da311b6ed-kube-api-access-vhv9v\") pod \"d5a2fe82-afb7-4822-b714-d44da311b6ed\" (UID: \"d5a2fe82-afb7-4822-b714-d44da311b6ed\") " Nov 02 23:36:37.258067 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:37.257635 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb5eb5ca-ea96-4366-b376-3abac4ce1d8a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb5eb5ca-ea96-4366-b376-3abac4ce1d8a-plugins-config-volume\") pod \"fb5eb5ca-ea96-4366-b376-3abac4ce1d8a\" (UID: \"fb5eb5ca-ea96-4366-b376-3abac4ce1d8a\") " Nov 02 23:36:37.258067 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:37.257683 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5a2fe82-afb7-4822-b714-d44da311b6ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5a2fe82-afb7-4822-b714-d44da311b6ed-plugins-config-volume\") pod \"d5a2fe82-afb7-4822-b714-d44da311b6ed\" (UID: \"d5a2fe82-afb7-4822-b714-d44da311b6ed\") " Nov 02 23:36:37.258067 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:37.257816 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb5eb5ca-ea96-4366-b376-3abac4ce1d8a-kube-api-access-mdnvk\" (UniqueName: \"kubernetes.io/projected/fb5eb5ca-ea96-4366-b376-3abac4ce1d8a-kube-api-access-mdnvk\") pod \"fb5eb5ca-ea96-4366-b376-3abac4ce1d8a\" (UID: \"fb5eb5ca-ea96-4366-b376-3abac4ce1d8a\") " Nov 02 23:36:37.258227 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:37.258090 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a2fe82-afb7-4822-b714-d44da311b6ed-plugins-config-volume" pod "d5a2fe82-afb7-4822-b714-d44da311b6ed" (UID: "d5a2fe82-afb7-4822-b714-d44da311b6ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:36:37.258227 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:37.258102 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb5eb5ca-ea96-4366-b376-3abac4ce1d8a-plugins-config-volume" pod "fb5eb5ca-ea96-4366-b376-3abac4ce1d8a" (UID: "fb5eb5ca-ea96-4366-b376-3abac4ce1d8a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:36:37.258326 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:37.258311 5076 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb5eb5ca-ea96-4366-b376-3abac4ce1d8a-plugins-config-volume\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:37.258373 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:37.258334 5076 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5a2fe82-afb7-4822-b714-d44da311b6ed-plugins-config-volume\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:37.259662 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:37.259607 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a2fe82-afb7-4822-b714-d44da311b6ed-kube-api-access-vhv9v" pod "d5a2fe82-afb7-4822-b714-d44da311b6ed" (UID: "d5a2fe82-afb7-4822-b714-d44da311b6ed"). InnerVolumeSpecName "kube-api-access-vhv9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:36:37.259760 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:37.259735 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb5eb5ca-ea96-4366-b376-3abac4ce1d8a-kube-api-access-mdnvk" pod "fb5eb5ca-ea96-4366-b376-3abac4ce1d8a" (UID: "fb5eb5ca-ea96-4366-b376-3abac4ce1d8a"). InnerVolumeSpecName "kube-api-access-mdnvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:36:37.372121 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:37.365631 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhv9v\" (UniqueName: \"kubernetes.io/projected/d5a2fe82-afb7-4822-b714-d44da311b6ed-kube-api-access-vhv9v\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:37.372121 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:37.365671 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdnvk\" (UniqueName: \"kubernetes.io/projected/fb5eb5ca-ea96-4366-b376-3abac4ce1d8a-kube-api-access-mdnvk\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:37.771915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:37.771753 5076 generic.go:356] "Generic (PLEG): container finished" podID="d5a2fe82-afb7-4822-b714-d44da311b6ed" containerID="501f4b1a7943ed29522d56ae1682af3b4b9c3e13938461db3e682e82ad03c260" exitCode=0 Nov 02 23:36:37.771915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:37.771883 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-981724-d5985fb4-s88v5" event={"ID":"d5a2fe82-afb7-4822-b714-d44da311b6ed","Type":"ContainerDied","Data":"501f4b1a7943ed29522d56ae1682af3b4b9c3e13938461db3e682e82ad03c260"} Nov 02 23:36:37.772186 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:37.771947 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-981724-d5985fb4-s88v5" event={"ID":"d5a2fe82-afb7-4822-b714-d44da311b6ed","Type":"ContainerDied","Data":"e6ce95a450bedc0d356a8f271118a4d91e7ec093a28b29205ba6e3d405f4f98c"} Nov 02 23:36:37.772186 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:37.771976 5076 scope.go:122] "RemoveContainer" containerID="501f4b1a7943ed29522d56ae1682af3b4b9c3e13938461db3e682e82ad03c260" Nov 02 23:36:37.772186 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:37.772080 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-981724-d5985fb4-s88v5" Nov 02 23:36:37.778473 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:37.778444 5076 generic.go:356] "Generic (PLEG): container finished" podID="fb5eb5ca-ea96-4366-b376-3abac4ce1d8a" containerID="263d8d00024e4886c14e6a4560721270211610b4fedb6619fdde5209378a6056" exitCode=0 Nov 02 23:36:37.778611 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:37.778506 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5bf989ccc5-mgxq9" Nov 02 23:36:37.778611 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:37.778521 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5bf989ccc5-mgxq9" event={"ID":"fb5eb5ca-ea96-4366-b376-3abac4ce1d8a","Type":"ContainerDied","Data":"263d8d00024e4886c14e6a4560721270211610b4fedb6619fdde5209378a6056"} Nov 02 23:36:37.778611 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:37.778577 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5bf989ccc5-mgxq9" event={"ID":"fb5eb5ca-ea96-4366-b376-3abac4ce1d8a","Type":"ContainerDied","Data":"9b0468e8f990a85da95a0deb2ad46d9f6e5b07a9b600a34f41ef3287fac70c98"} Nov 02 23:36:37.790422 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:37.790402 5076 scope.go:122] "RemoveContainer" containerID="501f4b1a7943ed29522d56ae1682af3b4b9c3e13938461db3e682e82ad03c260" Nov 02 23:36:37.790719 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:36:37.790697 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"501f4b1a7943ed29522d56ae1682af3b4b9c3e13938461db3e682e82ad03c260\": container with ID starting with 501f4b1a7943ed29522d56ae1682af3b4b9c3e13938461db3e682e82ad03c260 not found: ID does not exist" containerID="501f4b1a7943ed29522d56ae1682af3b4b9c3e13938461db3e682e82ad03c260" Nov 02 23:36:37.790779 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:37.790731 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"501f4b1a7943ed29522d56ae1682af3b4b9c3e13938461db3e682e82ad03c260"} err="failed to get container status \"501f4b1a7943ed29522d56ae1682af3b4b9c3e13938461db3e682e82ad03c260\": rpc error: code = NotFound desc = could not find container \"501f4b1a7943ed29522d56ae1682af3b4b9c3e13938461db3e682e82ad03c260\": container with ID starting with 501f4b1a7943ed29522d56ae1682af3b4b9c3e13938461db3e682e82ad03c260 not found: ID does not exist" Nov 02 23:36:37.790779 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:37.790749 5076 scope.go:122] "RemoveContainer" containerID="263d8d00024e4886c14e6a4560721270211610b4fedb6619fdde5209378a6056" Nov 02 23:36:37.799473 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:37.799448 5076 scope.go:122] "RemoveContainer" containerID="263d8d00024e4886c14e6a4560721270211610b4fedb6619fdde5209378a6056" Nov 02 23:36:37.799757 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:36:37.799735 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"263d8d00024e4886c14e6a4560721270211610b4fedb6619fdde5209378a6056\": container with ID starting with 263d8d00024e4886c14e6a4560721270211610b4fedb6619fdde5209378a6056 not found: ID does not exist" containerID="263d8d00024e4886c14e6a4560721270211610b4fedb6619fdde5209378a6056" Nov 02 23:36:37.799814 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:37.799765 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"263d8d00024e4886c14e6a4560721270211610b4fedb6619fdde5209378a6056"} err="failed to get container status \"263d8d00024e4886c14e6a4560721270211610b4fedb6619fdde5209378a6056\": rpc error: code = NotFound desc = could not find container \"263d8d00024e4886c14e6a4560721270211610b4fedb6619fdde5209378a6056\": container with ID starting with 263d8d00024e4886c14e6a4560721270211610b4fedb6619fdde5209378a6056 not found: ID does not exist" Nov 02 23:36:37.969512 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:37.969473 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-981724-d5985fb4-s88v5"] Nov 02 23:36:37.977525 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:37.977485 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-981724-d5985fb4-s88v5"] Nov 02 23:36:38.043793 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:38.043663 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5bf989ccc5-mgxq9"] Nov 02 23:36:38.049946 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:38.049884 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5bf989ccc5-mgxq9"] Nov 02 23:36:38.259692 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:38.259648 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5d87-5b879bb4b-4kh76" podUID="3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.149:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:36:38.615055 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:38.615019 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp"] Nov 02 23:36:38.615578 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:38.615541 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" podUID="86a56d46-242b-4863-8b05-4c5054fa0d90" containerName="istio-proxy" containerID="cri-o://6d52a3e9f0fbb68c9c4f419d3e730d2b9da8929c1e564d53d016710d0ae6b590" gracePeriod=30 Nov 02 23:36:39.147834 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:39.147749 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5a2fe82-afb7-4822-b714-d44da311b6ed" path="/var/lib/kubelet/pods/d5a2fe82-afb7-4822-b714-d44da311b6ed/volumes" Nov 02 23:36:39.148418 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:39.148397 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb5eb5ca-ea96-4366-b376-3abac4ce1d8a" path="/var/lib/kubelet/pods/fb5eb5ca-ea96-4366-b376-3abac4ce1d8a/volumes" Nov 02 23:36:39.887267 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:36:39.887232 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:36:43.885960 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:43.883241 5076 generic.go:356] "Generic (PLEG): container finished" podID="86a56d46-242b-4863-8b05-4c5054fa0d90" containerID="6d52a3e9f0fbb68c9c4f419d3e730d2b9da8929c1e564d53d016710d0ae6b590" exitCode=0 Nov 02 23:36:43.885960 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:43.883358 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" event={"ID":"86a56d46-242b-4863-8b05-4c5054fa0d90","Type":"ContainerDied","Data":"6d52a3e9f0fbb68c9c4f419d3e730d2b9da8929c1e564d53d016710d0ae6b590"} Nov 02 23:36:43.885960 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:43.883405 5076 scope.go:122] "RemoveContainer" containerID="e636865b78586e81385059a5a941351a8e90000c6b818967049bdcd15cceb128" Nov 02 23:36:43.953988 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:43.953961 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" Nov 02 23:36:44.075747 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:44.075712 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86a56d46-242b-4863-8b05-4c5054fa0d90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a56d46-242b-4863-8b05-4c5054fa0d90-workload-socket\") pod \"86a56d46-242b-4863-8b05-4c5054fa0d90\" (UID: \"86a56d46-242b-4863-8b05-4c5054fa0d90\") " Nov 02 23:36:44.075951 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:44.075777 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86a56d46-242b-4863-8b05-4c5054fa0d90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a56d46-242b-4863-8b05-4c5054fa0d90-credential-socket\") pod \"86a56d46-242b-4863-8b05-4c5054fa0d90\" (UID: \"86a56d46-242b-4863-8b05-4c5054fa0d90\") " Nov 02 23:36:44.075951 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:44.075844 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86a56d46-242b-4863-8b05-4c5054fa0d90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86a56d46-242b-4863-8b05-4c5054fa0d90-istiod-ca-cert\") pod \"86a56d46-242b-4863-8b05-4c5054fa0d90\" (UID: \"86a56d46-242b-4863-8b05-4c5054fa0d90\") " Nov 02 23:36:44.075951 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:44.075872 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86a56d46-242b-4863-8b05-4c5054fa0d90-kube-api-access-9wssw\" (UniqueName: \"kubernetes.io/projected/86a56d46-242b-4863-8b05-4c5054fa0d90-kube-api-access-9wssw\") pod \"86a56d46-242b-4863-8b05-4c5054fa0d90\" (UID: \"86a56d46-242b-4863-8b05-4c5054fa0d90\") " Nov 02 23:36:44.075951 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:44.075943 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86a56d46-242b-4863-8b05-4c5054fa0d90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86a56d46-242b-4863-8b05-4c5054fa0d90-istio-podinfo\") pod \"86a56d46-242b-4863-8b05-4c5054fa0d90\" (UID: \"86a56d46-242b-4863-8b05-4c5054fa0d90\") " Nov 02 23:36:44.076193 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:44.076018 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86a56d46-242b-4863-8b05-4c5054fa0d90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86a56d46-242b-4863-8b05-4c5054fa0d90-istio-envoy\") pod \"86a56d46-242b-4863-8b05-4c5054fa0d90\" (UID: \"86a56d46-242b-4863-8b05-4c5054fa0d90\") " Nov 02 23:36:44.076193 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:44.076061 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86a56d46-242b-4863-8b05-4c5054fa0d90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86a56d46-242b-4863-8b05-4c5054fa0d90-workload-certs\") pod \"86a56d46-242b-4863-8b05-4c5054fa0d90\" (UID: \"86a56d46-242b-4863-8b05-4c5054fa0d90\") " Nov 02 23:36:44.076193 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:44.076091 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86a56d46-242b-4863-8b05-4c5054fa0d90-istio-token\" (UniqueName: \"kubernetes.io/projected/86a56d46-242b-4863-8b05-4c5054fa0d90-istio-token\") pod \"86a56d46-242b-4863-8b05-4c5054fa0d90\" (UID: \"86a56d46-242b-4863-8b05-4c5054fa0d90\") " Nov 02 23:36:44.076193 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:44.076088 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86a56d46-242b-4863-8b05-4c5054fa0d90-workload-socket" pod "86a56d46-242b-4863-8b05-4c5054fa0d90" (UID: "86a56d46-242b-4863-8b05-4c5054fa0d90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:36:44.076193 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:44.076142 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86a56d46-242b-4863-8b05-4c5054fa0d90-credential-socket" pod "86a56d46-242b-4863-8b05-4c5054fa0d90" (UID: "86a56d46-242b-4863-8b05-4c5054fa0d90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:36:44.076193 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:44.076167 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86a56d46-242b-4863-8b05-4c5054fa0d90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86a56d46-242b-4863-8b05-4c5054fa0d90-istio-data\") pod \"86a56d46-242b-4863-8b05-4c5054fa0d90\" (UID: \"86a56d46-242b-4863-8b05-4c5054fa0d90\") " Nov 02 23:36:44.076504 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:44.076331 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86a56d46-242b-4863-8b05-4c5054fa0d90-workload-certs" pod "86a56d46-242b-4863-8b05-4c5054fa0d90" (UID: "86a56d46-242b-4863-8b05-4c5054fa0d90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:36:44.076582 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:44.076561 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86a56d46-242b-4863-8b05-4c5054fa0d90-istiod-ca-cert" pod "86a56d46-242b-4863-8b05-4c5054fa0d90" (UID: "86a56d46-242b-4863-8b05-4c5054fa0d90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:36:44.076687 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:44.076655 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86a56d46-242b-4863-8b05-4c5054fa0d90-istio-data" pod "86a56d46-242b-4863-8b05-4c5054fa0d90" (UID: "86a56d46-242b-4863-8b05-4c5054fa0d90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:36:44.078564 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:44.078526 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86a56d46-242b-4863-8b05-4c5054fa0d90-kube-api-access-9wssw" pod "86a56d46-242b-4863-8b05-4c5054fa0d90" (UID: "86a56d46-242b-4863-8b05-4c5054fa0d90"). InnerVolumeSpecName "kube-api-access-9wssw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:36:44.078791 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:44.078549 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86a56d46-242b-4863-8b05-4c5054fa0d90-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:44.078791 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:44.078595 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86a56d46-242b-4863-8b05-4c5054fa0d90-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:44.078791 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:44.078614 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86a56d46-242b-4863-8b05-4c5054fa0d90-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:44.078791 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:44.078628 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a56d46-242b-4863-8b05-4c5054fa0d90-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:44.078791 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:44.078642 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a56d46-242b-4863-8b05-4c5054fa0d90-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:44.079082 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:44.078867 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86a56d46-242b-4863-8b05-4c5054fa0d90-istio-envoy" pod "86a56d46-242b-4863-8b05-4c5054fa0d90" (UID: "86a56d46-242b-4863-8b05-4c5054fa0d90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:36:44.080554 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:44.080533 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86a56d46-242b-4863-8b05-4c5054fa0d90-istio-token" pod "86a56d46-242b-4863-8b05-4c5054fa0d90" (UID: "86a56d46-242b-4863-8b05-4c5054fa0d90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:36:44.080686 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:44.080592 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86a56d46-242b-4863-8b05-4c5054fa0d90-istio-podinfo" pod "86a56d46-242b-4863-8b05-4c5054fa0d90" (UID: "86a56d46-242b-4863-8b05-4c5054fa0d90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:36:44.181367 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:44.181266 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wssw\" (UniqueName: \"kubernetes.io/projected/86a56d46-242b-4863-8b05-4c5054fa0d90-kube-api-access-9wssw\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:44.181630 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:44.181611 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86a56d46-242b-4863-8b05-4c5054fa0d90-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:44.181747 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:44.181734 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86a56d46-242b-4863-8b05-4c5054fa0d90-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:44.181836 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:44.181822 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86a56d46-242b-4863-8b05-4c5054fa0d90-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:44.911913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:44.911645 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" event={"ID":"86a56d46-242b-4863-8b05-4c5054fa0d90","Type":"ContainerDied","Data":"1dffe92e8bb3493427a30c07b0e6aba77f683c95b7ea23d3cecb96ed77dad895"} Nov 02 23:36:44.911913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:44.911739 5076 scope.go:122] "RemoveContainer" containerID="6d52a3e9f0fbb68c9c4f419d3e730d2b9da8929c1e564d53d016710d0ae6b590" Nov 02 23:36:44.911913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:44.911754 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp" Nov 02 23:36:45.117471 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:45.117423 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp"] Nov 02 23:36:45.164614 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:45.164521 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-981724-openshift-default-744ccf4df-k4rsp"] Nov 02 23:36:47.147784 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:47.147750 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86a56d46-242b-4863-8b05-4c5054fa0d90" path="/var/lib/kubelet/pods/86a56d46-242b-4863-8b05-4c5054fa0d90/volumes" Nov 02 23:36:50.008936 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.008877 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x"] Nov 02 23:36:50.009420 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.009406 5076 cpu_manager.go:414] "containerMap: removing container" podUID="fb5eb5ca-ea96-4366-b376-3abac4ce1d8a" containerName="payload-pre-processing" Nov 02 23:36:50.009478 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.009421 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb5eb5ca-ea96-4366-b376-3abac4ce1d8a" containerName="payload-pre-processing" Nov 02 23:36:50.009478 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.009433 5076 cpu_manager.go:414] "containerMap: removing container" podUID="3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766" containerName="maas-api" Nov 02 23:36:50.009478 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.009438 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766" containerName="maas-api" Nov 02 23:36:50.009478 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.009447 5076 cpu_manager.go:414] "containerMap: removing container" podUID="8895d1e5-9514-4125-8343-e8d3c9cf3d09" containerName="payload-pre-processing" Nov 02 23:36:50.009478 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.009452 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8895d1e5-9514-4125-8343-e8d3c9cf3d09" containerName="payload-pre-processing" Nov 02 23:36:50.009478 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.009465 5076 cpu_manager.go:414] "containerMap: removing container" podUID="e02e672a-a973-4706-9c2b-a307aabafa9b" containerName="payload-processing" Nov 02 23:36:50.009478 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.009471 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e02e672a-a973-4706-9c2b-a307aabafa9b" containerName="payload-processing" Nov 02 23:36:50.009478 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.009481 5076 cpu_manager.go:414] "containerMap: removing container" podUID="8bcab402-b870-4284-9d80-a7c8243492bd" containerName="istio-proxy" Nov 02 23:36:50.009742 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.009486 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bcab402-b870-4284-9d80-a7c8243492bd" containerName="istio-proxy" Nov 02 23:36:50.009742 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.009497 5076 cpu_manager.go:414] "containerMap: removing container" podUID="8bcab402-b870-4284-9d80-a7c8243492bd" containerName="istio-proxy" Nov 02 23:36:50.009742 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.009502 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bcab402-b870-4284-9d80-a7c8243492bd" containerName="istio-proxy" Nov 02 23:36:50.009742 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.009517 5076 cpu_manager.go:414] "containerMap: removing container" podUID="d5a2fe82-afb7-4822-b714-d44da311b6ed" containerName="payload-processing" Nov 02 23:36:50.009742 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.009521 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5a2fe82-afb7-4822-b714-d44da311b6ed" containerName="payload-processing" Nov 02 23:36:50.009742 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.009532 5076 cpu_manager.go:414] "containerMap: removing container" podUID="145c9784-8db4-4686-b77d-7f33d51272d0" containerName="maas-api" Nov 02 23:36:50.009742 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.009537 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="145c9784-8db4-4686-b77d-7f33d51272d0" containerName="maas-api" Nov 02 23:36:50.009742 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.009548 5076 cpu_manager.go:414] "containerMap: removing container" podUID="86a56d46-242b-4863-8b05-4c5054fa0d90" containerName="istio-proxy" Nov 02 23:36:50.009742 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.009553 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86a56d46-242b-4863-8b05-4c5054fa0d90" containerName="istio-proxy" Nov 02 23:36:50.009742 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.009627 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb5eb5ca-ea96-4366-b376-3abac4ce1d8a" containerName="payload-pre-processing" Nov 02 23:36:50.009742 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.009635 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c1c8cf5-4bc1-441f-b0f7-5382d0ab5766" containerName="maas-api" Nov 02 23:36:50.009742 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.009642 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="8895d1e5-9514-4125-8343-e8d3c9cf3d09" containerName="payload-pre-processing" Nov 02 23:36:50.009742 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.009648 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="e02e672a-a973-4706-9c2b-a307aabafa9b" containerName="payload-processing" Nov 02 23:36:50.009742 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.009654 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bcab402-b870-4284-9d80-a7c8243492bd" containerName="istio-proxy" Nov 02 23:36:50.009742 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.009660 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bcab402-b870-4284-9d80-a7c8243492bd" containerName="istio-proxy" Nov 02 23:36:50.009742 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.009668 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5a2fe82-afb7-4822-b714-d44da311b6ed" containerName="payload-processing" Nov 02 23:36:50.009742 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.009674 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="145c9784-8db4-4686-b77d-7f33d51272d0" containerName="maas-api" Nov 02 23:36:50.009742 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.009679 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="86a56d46-242b-4863-8b05-4c5054fa0d90" containerName="istio-proxy" Nov 02 23:36:50.009742 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.009687 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="86a56d46-242b-4863-8b05-4c5054fa0d90" containerName="istio-proxy" Nov 02 23:36:50.014856 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.014836 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" Nov 02 23:36:50.110935 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.104010 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd88117-cf15-4806-ade9-c17982846d7d-credential-socket\") pod \"e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x\" (UID: \"3fd88117-cf15-4806-ade9-c17982846d7d\") " pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" Nov 02 23:36:50.110935 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.104073 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fd88117-cf15-4806-ade9-c17982846d7d-istio-token\") pod \"e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x\" (UID: \"3fd88117-cf15-4806-ade9-c17982846d7d\") " pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" Nov 02 23:36:50.110935 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.104122 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fd88117-cf15-4806-ade9-c17982846d7d-istio-podinfo\") pod \"e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x\" (UID: \"3fd88117-cf15-4806-ade9-c17982846d7d\") " pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" Nov 02 23:36:50.110935 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.104187 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fd88117-cf15-4806-ade9-c17982846d7d-istio-envoy\") pod \"e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x\" (UID: \"3fd88117-cf15-4806-ade9-c17982846d7d\") " pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" Nov 02 23:36:50.110935 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.104217 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8thrr\" (UniqueName: \"kubernetes.io/projected/3fd88117-cf15-4806-ade9-c17982846d7d-kube-api-access-8thrr\") pod \"e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x\" (UID: \"3fd88117-cf15-4806-ade9-c17982846d7d\") " pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" Nov 02 23:36:50.110935 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.104296 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fd88117-cf15-4806-ade9-c17982846d7d-workload-certs\") pod \"e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x\" (UID: \"3fd88117-cf15-4806-ade9-c17982846d7d\") " pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" Nov 02 23:36:50.110935 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.104340 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fd88117-cf15-4806-ade9-c17982846d7d-istio-data\") pod \"e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x\" (UID: \"3fd88117-cf15-4806-ade9-c17982846d7d\") " pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" Nov 02 23:36:50.110935 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.104385 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd88117-cf15-4806-ade9-c17982846d7d-workload-socket\") pod \"e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x\" (UID: \"3fd88117-cf15-4806-ade9-c17982846d7d\") " pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" Nov 02 23:36:50.110935 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.104420 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fd88117-cf15-4806-ade9-c17982846d7d-istiod-ca-cert\") pod \"e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x\" (UID: \"3fd88117-cf15-4806-ade9-c17982846d7d\") " pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" Nov 02 23:36:50.117137 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.117102 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-9672a1-openshift-default-dockercfg-8d9x6\"" Nov 02 23:36:50.121617 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.121585 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x"] Nov 02 23:36:50.217911 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.217063 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fd88117-cf15-4806-ade9-c17982846d7d-istio-data\") pod \"e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x\" (UID: \"3fd88117-cf15-4806-ade9-c17982846d7d\") " pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" Nov 02 23:36:50.217911 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.217553 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd88117-cf15-4806-ade9-c17982846d7d-workload-socket\") pod \"e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x\" (UID: \"3fd88117-cf15-4806-ade9-c17982846d7d\") " pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" Nov 02 23:36:50.217911 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.217593 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fd88117-cf15-4806-ade9-c17982846d7d-istiod-ca-cert\") pod \"e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x\" (UID: \"3fd88117-cf15-4806-ade9-c17982846d7d\") " pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" Nov 02 23:36:50.217911 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.217650 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd88117-cf15-4806-ade9-c17982846d7d-credential-socket\") pod \"e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x\" (UID: \"3fd88117-cf15-4806-ade9-c17982846d7d\") " pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" Nov 02 23:36:50.217911 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.217687 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fd88117-cf15-4806-ade9-c17982846d7d-istio-token\") pod \"e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x\" (UID: \"3fd88117-cf15-4806-ade9-c17982846d7d\") " pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" Nov 02 23:36:50.217911 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.217754 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fd88117-cf15-4806-ade9-c17982846d7d-istio-podinfo\") pod \"e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x\" (UID: \"3fd88117-cf15-4806-ade9-c17982846d7d\") " pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" Nov 02 23:36:50.217911 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.217841 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fd88117-cf15-4806-ade9-c17982846d7d-istio-envoy\") pod \"e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x\" (UID: \"3fd88117-cf15-4806-ade9-c17982846d7d\") " pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" Nov 02 23:36:50.217911 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.217871 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8thrr\" (UniqueName: \"kubernetes.io/projected/3fd88117-cf15-4806-ade9-c17982846d7d-kube-api-access-8thrr\") pod \"e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x\" (UID: \"3fd88117-cf15-4806-ade9-c17982846d7d\") " pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" Nov 02 23:36:50.218492 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.217989 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fd88117-cf15-4806-ade9-c17982846d7d-workload-certs\") pod \"e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x\" (UID: \"3fd88117-cf15-4806-ade9-c17982846d7d\") " pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" Nov 02 23:36:50.218492 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.218414 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fd88117-cf15-4806-ade9-c17982846d7d-workload-certs\") pod \"e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x\" (UID: \"3fd88117-cf15-4806-ade9-c17982846d7d\") " pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" Nov 02 23:36:50.218876 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.218671 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd88117-cf15-4806-ade9-c17982846d7d-workload-socket\") pod \"e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x\" (UID: \"3fd88117-cf15-4806-ade9-c17982846d7d\") " pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" Nov 02 23:36:50.219761 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.219282 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fd88117-cf15-4806-ade9-c17982846d7d-istio-data\") pod \"e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x\" (UID: \"3fd88117-cf15-4806-ade9-c17982846d7d\") " pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" Nov 02 23:36:50.219761 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.219328 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fd88117-cf15-4806-ade9-c17982846d7d-istiod-ca-cert\") pod \"e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x\" (UID: \"3fd88117-cf15-4806-ade9-c17982846d7d\") " pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" Nov 02 23:36:50.219761 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.219756 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd88117-cf15-4806-ade9-c17982846d7d-credential-socket\") pod \"e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x\" (UID: \"3fd88117-cf15-4806-ade9-c17982846d7d\") " pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" Nov 02 23:36:50.222804 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.222775 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fd88117-cf15-4806-ade9-c17982846d7d-istio-envoy\") pod \"e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x\" (UID: \"3fd88117-cf15-4806-ade9-c17982846d7d\") " pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" Nov 02 23:36:50.234255 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.232412 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fd88117-cf15-4806-ade9-c17982846d7d-istio-podinfo\") pod \"e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x\" (UID: \"3fd88117-cf15-4806-ade9-c17982846d7d\") " pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" Nov 02 23:36:50.273542 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.273474 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8thrr\" (UniqueName: \"kubernetes.io/projected/3fd88117-cf15-4806-ade9-c17982846d7d-kube-api-access-8thrr\") pod \"e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x\" (UID: \"3fd88117-cf15-4806-ade9-c17982846d7d\") " pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" Nov 02 23:36:50.276475 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.276446 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fd88117-cf15-4806-ade9-c17982846d7d-istio-token\") pod \"e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x\" (UID: \"3fd88117-cf15-4806-ade9-c17982846d7d\") " pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" Nov 02 23:36:50.331058 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.330312 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" Nov 02 23:36:50.772419 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.772392 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x"] Nov 02 23:36:50.773961 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:36:50.773933 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fd88117_cf15_4806_ade9_c17982846d7d.slice/crio-dee5ca8dbf60f79b7ac87479cfb742018ef3037261c7ac8853787ec624999bc8 WatchSource:0}: Error finding container dee5ca8dbf60f79b7ac87479cfb742018ef3037261c7ac8853787ec624999bc8: Status 404 returned error can't find the container with id dee5ca8dbf60f79b7ac87479cfb742018ef3037261c7ac8853787ec624999bc8 Nov 02 23:36:50.776471 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.776432 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:36:50.776586 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.776512 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:36:50.776586 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.776545 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:36:50.877341 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.877306 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5"] Nov 02 23:36:50.877920 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.877877 5076 cpu_manager.go:414] "containerMap: removing container" podUID="86a56d46-242b-4863-8b05-4c5054fa0d90" containerName="istio-proxy" Nov 02 23:36:50.877920 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.877906 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86a56d46-242b-4863-8b05-4c5054fa0d90" containerName="istio-proxy" Nov 02 23:36:50.881470 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.881447 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" Nov 02 23:36:50.940240 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.940204 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5"] Nov 02 23:36:50.969710 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.969661 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" event={"ID":"3fd88117-cf15-4806-ade9-c17982846d7d","Type":"ContainerStarted","Data":"b806b7dfdb2b05430ae52c431243adad3d9c999fc29dac4f7e47874c6a5db367"} Nov 02 23:36:50.969882 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:50.969735 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" event={"ID":"3fd88117-cf15-4806-ade9-c17982846d7d","Type":"ContainerStarted","Data":"dee5ca8dbf60f79b7ac87479cfb742018ef3037261c7ac8853787ec624999bc8"} Nov 02 23:36:51.041594 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:51.041495 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db169a4-82e1-4ed5-b64c-adb40cab5629-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5\" (UID: \"4db169a4-82e1-4ed5-b64c-adb40cab5629\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" Nov 02 23:36:51.041594 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:51.041535 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdt2p\" (UniqueName: \"kubernetes.io/projected/4db169a4-82e1-4ed5-b64c-adb40cab5629-kube-api-access-mdt2p\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5\" (UID: \"4db169a4-82e1-4ed5-b64c-adb40cab5629\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" Nov 02 23:36:51.042155 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:51.041692 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4db169a4-82e1-4ed5-b64c-adb40cab5629-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5\" (UID: \"4db169a4-82e1-4ed5-b64c-adb40cab5629\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" Nov 02 23:36:51.042155 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:51.041745 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4db169a4-82e1-4ed5-b64c-adb40cab5629-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5\" (UID: \"4db169a4-82e1-4ed5-b64c-adb40cab5629\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" Nov 02 23:36:51.042155 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:51.041837 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4db169a4-82e1-4ed5-b64c-adb40cab5629-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5\" (UID: \"4db169a4-82e1-4ed5-b64c-adb40cab5629\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" Nov 02 23:36:51.042155 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:51.041863 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4db169a4-82e1-4ed5-b64c-adb40cab5629-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5\" (UID: \"4db169a4-82e1-4ed5-b64c-adb40cab5629\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" Nov 02 23:36:51.042155 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:51.042012 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db169a4-82e1-4ed5-b64c-adb40cab5629-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5\" (UID: \"4db169a4-82e1-4ed5-b64c-adb40cab5629\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" Nov 02 23:36:51.042155 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:51.042044 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4db169a4-82e1-4ed5-b64c-adb40cab5629-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5\" (UID: \"4db169a4-82e1-4ed5-b64c-adb40cab5629\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" Nov 02 23:36:51.042357 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:51.042183 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4db169a4-82e1-4ed5-b64c-adb40cab5629-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5\" (UID: \"4db169a4-82e1-4ed5-b64c-adb40cab5629\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" Nov 02 23:36:51.105060 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:51.105007 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" podStartSLOduration=2.104993526 podStartE2EDuration="2.104993526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:36:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:36:51.10313044 +0000 UTC m=+2660.071662224" watchObservedRunningTime="2026-11-02 23:36:51.104993526 +0000 UTC m=+2660.073525309" Nov 02 23:36:51.143092 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:51.143044 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4db169a4-82e1-4ed5-b64c-adb40cab5629-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5\" (UID: \"4db169a4-82e1-4ed5-b64c-adb40cab5629\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" Nov 02 23:36:51.143280 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:51.143158 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db169a4-82e1-4ed5-b64c-adb40cab5629-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5\" (UID: \"4db169a4-82e1-4ed5-b64c-adb40cab5629\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" Nov 02 23:36:51.143280 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:51.143187 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdt2p\" (UniqueName: \"kubernetes.io/projected/4db169a4-82e1-4ed5-b64c-adb40cab5629-kube-api-access-mdt2p\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5\" (UID: \"4db169a4-82e1-4ed5-b64c-adb40cab5629\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" Nov 02 23:36:51.143280 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:51.143257 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4db169a4-82e1-4ed5-b64c-adb40cab5629-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5\" (UID: \"4db169a4-82e1-4ed5-b64c-adb40cab5629\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" Nov 02 23:36:51.143464 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:51.143294 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4db169a4-82e1-4ed5-b64c-adb40cab5629-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5\" (UID: \"4db169a4-82e1-4ed5-b64c-adb40cab5629\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" Nov 02 23:36:51.143464 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:51.143327 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4db169a4-82e1-4ed5-b64c-adb40cab5629-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5\" (UID: \"4db169a4-82e1-4ed5-b64c-adb40cab5629\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" Nov 02 23:36:51.143464 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:51.143349 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4db169a4-82e1-4ed5-b64c-adb40cab5629-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5\" (UID: \"4db169a4-82e1-4ed5-b64c-adb40cab5629\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" Nov 02 23:36:51.143638 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:51.143471 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db169a4-82e1-4ed5-b64c-adb40cab5629-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5\" (UID: \"4db169a4-82e1-4ed5-b64c-adb40cab5629\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" Nov 02 23:36:51.143638 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:51.143500 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4db169a4-82e1-4ed5-b64c-adb40cab5629-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5\" (UID: \"4db169a4-82e1-4ed5-b64c-adb40cab5629\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" Nov 02 23:36:51.143912 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:51.143861 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db169a4-82e1-4ed5-b64c-adb40cab5629-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5\" (UID: \"4db169a4-82e1-4ed5-b64c-adb40cab5629\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" Nov 02 23:36:51.144045 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:51.144024 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4db169a4-82e1-4ed5-b64c-adb40cab5629-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5\" (UID: \"4db169a4-82e1-4ed5-b64c-adb40cab5629\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" Nov 02 23:36:51.144226 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:51.144197 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4db169a4-82e1-4ed5-b64c-adb40cab5629-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5\" (UID: \"4db169a4-82e1-4ed5-b64c-adb40cab5629\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" Nov 02 23:36:51.144367 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:51.144330 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db169a4-82e1-4ed5-b64c-adb40cab5629-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5\" (UID: \"4db169a4-82e1-4ed5-b64c-adb40cab5629\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" Nov 02 23:36:51.144584 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:51.144560 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4db169a4-82e1-4ed5-b64c-adb40cab5629-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5\" (UID: \"4db169a4-82e1-4ed5-b64c-adb40cab5629\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" Nov 02 23:36:51.145875 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:51.145852 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4db169a4-82e1-4ed5-b64c-adb40cab5629-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5\" (UID: \"4db169a4-82e1-4ed5-b64c-adb40cab5629\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" Nov 02 23:36:51.146480 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:51.146455 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4db169a4-82e1-4ed5-b64c-adb40cab5629-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5\" (UID: \"4db169a4-82e1-4ed5-b64c-adb40cab5629\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" Nov 02 23:36:51.198441 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:51.198408 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdt2p\" (UniqueName: \"kubernetes.io/projected/4db169a4-82e1-4ed5-b64c-adb40cab5629-kube-api-access-mdt2p\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5\" (UID: \"4db169a4-82e1-4ed5-b64c-adb40cab5629\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" Nov 02 23:36:51.198592 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:51.198475 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4db169a4-82e1-4ed5-b64c-adb40cab5629-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5\" (UID: \"4db169a4-82e1-4ed5-b64c-adb40cab5629\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" Nov 02 23:36:51.330993 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:51.330882 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" Nov 02 23:36:51.336830 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:51.336801 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" Nov 02 23:36:51.497631 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:51.497600 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" Nov 02 23:36:51.918802 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:51.918763 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5"] Nov 02 23:36:51.924390 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:36:51.924360 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4db169a4_82e1_4ed5_b64c_adb40cab5629.slice/crio-0c6c86b977337c36e057996f9d8b05af4e179ea1733185670455645a80c47dda WatchSource:0}: Error finding container 0c6c86b977337c36e057996f9d8b05af4e179ea1733185670455645a80c47dda: Status 404 returned error can't find the container with id 0c6c86b977337c36e057996f9d8b05af4e179ea1733185670455645a80c47dda Nov 02 23:36:51.927064 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:51.927036 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:36:51.927154 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:51.927102 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:36:51.927154 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:51.927131 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:36:51.978749 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:51.978719 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" event={"ID":"4db169a4-82e1-4ed5-b64c-adb40cab5629","Type":"ContainerStarted","Data":"0c6c86b977337c36e057996f9d8b05af4e179ea1733185670455645a80c47dda"} Nov 02 23:36:51.979011 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:51.978991 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" Nov 02 23:36:51.980050 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:51.980029 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" Nov 02 23:36:52.987276 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:52.987241 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" event={"ID":"4db169a4-82e1-4ed5-b64c-adb40cab5629","Type":"ContainerStarted","Data":"f5165fb81e18cb08871e6a5af1ac049144a79bf53edbf7895178c3bb0e52048a"} Nov 02 23:36:53.498705 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:53.498674 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" Nov 02 23:36:53.500067 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:53.500037 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 02 23:36:53.500186 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:53.500102 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" podUID="4db169a4-82e1-4ed5-b64c-adb40cab5629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 02 23:36:54.498308 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:54.498268 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 02 23:36:54.498740 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:54.498326 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" podUID="4db169a4-82e1-4ed5-b64c-adb40cab5629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 02 23:36:55.498404 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:55.498369 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 02 23:36:55.498810 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:55.498422 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" podUID="4db169a4-82e1-4ed5-b64c-adb40cab5629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 02 23:36:56.499050 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:56.499012 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 02 23:36:56.499595 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:56.499069 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" podUID="4db169a4-82e1-4ed5-b64c-adb40cab5629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 02 23:36:57.498279 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:57.498248 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 02 23:36:57.498437 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:57.498299 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" podUID="4db169a4-82e1-4ed5-b64c-adb40cab5629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 02 23:36:58.498579 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:58.498543 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 02 23:36:58.499026 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:58.498592 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" podUID="4db169a4-82e1-4ed5-b64c-adb40cab5629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 02 23:36:59.729708 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:59.729674 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" Nov 02 23:36:59.730251 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:59.730228 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" Nov 02 23:36:59.731710 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:59.731688 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" Nov 02 23:36:59.888730 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:36:59.888679 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" podStartSLOduration=9.88866751 podStartE2EDuration="9.88866751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:36:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:36:53.192409397 +0000 UTC m=+2662.160941182" watchObservedRunningTime="2026-11-02 23:36:59.88866751 +0000 UTC m=+2668.857199293" Nov 02 23:37:09.883807 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:37:09.883765 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:37:28.359733 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:28.359694 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5ff336-7d447bd878-v7jmb"] Nov 02 23:37:28.360367 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:28.359946 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-5ff336-7d447bd878-v7jmb" podUID="89a8789b-2ff7-4c73-9bee-2d3879cbe2b9" containerName="payload-processing" containerID="cri-o://eaf11421bb0c333736917c51669b4f46a8c723acfc901c22d7529dd49bb22b7b" gracePeriod=30 Nov 02 23:37:28.464150 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:28.464111 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-5fdb8d5c9f-8xdhn"] Nov 02 23:37:28.464392 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:28.464363 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-5fdb8d5c9f-8xdhn" podUID="a6ef156b-92aa-4950-b83e-294e4adc2e77" containerName="payload-pre-processing" containerID="cri-o://fa0b7f5ace5dd7309266db5f2ebd09b16fb0911167827e14bee90bc76bfae09c" gracePeriod=30 Nov 02 23:37:28.651300 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:28.651272 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5ff336-7d447bd878-v7jmb" Nov 02 23:37:28.684664 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:28.684617 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89a8789b-2ff7-4c73-9bee-2d3879cbe2b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a8789b-2ff7-4c73-9bee-2d3879cbe2b9-plugins-config-volume\") pod \"89a8789b-2ff7-4c73-9bee-2d3879cbe2b9\" (UID: \"89a8789b-2ff7-4c73-9bee-2d3879cbe2b9\") " Nov 02 23:37:28.684851 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:28.684678 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89a8789b-2ff7-4c73-9bee-2d3879cbe2b9-kube-api-access-xtxlp\" (UniqueName: \"kubernetes.io/projected/89a8789b-2ff7-4c73-9bee-2d3879cbe2b9-kube-api-access-xtxlp\") pod \"89a8789b-2ff7-4c73-9bee-2d3879cbe2b9\" (UID: \"89a8789b-2ff7-4c73-9bee-2d3879cbe2b9\") " Nov 02 23:37:28.685078 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:28.685049 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89a8789b-2ff7-4c73-9bee-2d3879cbe2b9-plugins-config-volume" pod "89a8789b-2ff7-4c73-9bee-2d3879cbe2b9" (UID: "89a8789b-2ff7-4c73-9bee-2d3879cbe2b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:37:28.685421 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:28.685376 5076 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a8789b-2ff7-4c73-9bee-2d3879cbe2b9-plugins-config-volume\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:28.687326 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:28.687301 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89a8789b-2ff7-4c73-9bee-2d3879cbe2b9-kube-api-access-xtxlp" pod "89a8789b-2ff7-4c73-9bee-2d3879cbe2b9" (UID: "89a8789b-2ff7-4c73-9bee-2d3879cbe2b9"). InnerVolumeSpecName "kube-api-access-xtxlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:37:28.720391 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:28.720366 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-5fdb8d5c9f-8xdhn" Nov 02 23:37:28.786407 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:28.786372 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6ef156b-92aa-4950-b83e-294e4adc2e77-kube-api-access-w47jg\" (UniqueName: \"kubernetes.io/projected/a6ef156b-92aa-4950-b83e-294e4adc2e77-kube-api-access-w47jg\") pod \"a6ef156b-92aa-4950-b83e-294e4adc2e77\" (UID: \"a6ef156b-92aa-4950-b83e-294e4adc2e77\") " Nov 02 23:37:28.786601 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:28.786534 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6ef156b-92aa-4950-b83e-294e4adc2e77-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6ef156b-92aa-4950-b83e-294e4adc2e77-plugins-config-volume\") pod \"a6ef156b-92aa-4950-b83e-294e4adc2e77\" (UID: \"a6ef156b-92aa-4950-b83e-294e4adc2e77\") " Nov 02 23:37:28.786944 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:28.786926 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtxlp\" (UniqueName: \"kubernetes.io/projected/89a8789b-2ff7-4c73-9bee-2d3879cbe2b9-kube-api-access-xtxlp\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:28.787037 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:28.786973 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6ef156b-92aa-4950-b83e-294e4adc2e77-plugins-config-volume" pod "a6ef156b-92aa-4950-b83e-294e4adc2e77" (UID: "a6ef156b-92aa-4950-b83e-294e4adc2e77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:37:28.788441 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:28.788419 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6ef156b-92aa-4950-b83e-294e4adc2e77-kube-api-access-w47jg" pod "a6ef156b-92aa-4950-b83e-294e4adc2e77" (UID: "a6ef156b-92aa-4950-b83e-294e4adc2e77"). InnerVolumeSpecName "kube-api-access-w47jg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:37:28.891971 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:28.890951 5076 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6ef156b-92aa-4950-b83e-294e4adc2e77-plugins-config-volume\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:28.891971 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:28.890987 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w47jg\" (UniqueName: \"kubernetes.io/projected/a6ef156b-92aa-4950-b83e-294e4adc2e77-kube-api-access-w47jg\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:29.342343 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:29.342304 5076 generic.go:356] "Generic (PLEG): container finished" podID="a6ef156b-92aa-4950-b83e-294e4adc2e77" containerID="fa0b7f5ace5dd7309266db5f2ebd09b16fb0911167827e14bee90bc76bfae09c" exitCode=0 Nov 02 23:37:29.342343 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:29.342341 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-5fdb8d5c9f-8xdhn" Nov 02 23:37:29.342596 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:29.342385 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-5fdb8d5c9f-8xdhn" event={"ID":"a6ef156b-92aa-4950-b83e-294e4adc2e77","Type":"ContainerDied","Data":"fa0b7f5ace5dd7309266db5f2ebd09b16fb0911167827e14bee90bc76bfae09c"} Nov 02 23:37:29.342596 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:29.342437 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-5fdb8d5c9f-8xdhn" event={"ID":"a6ef156b-92aa-4950-b83e-294e4adc2e77","Type":"ContainerDied","Data":"f03aa0a8e05df12b5cf7ecae8e87dbac7ecaf4332737c9a1ea75b4c8dd49eea6"} Nov 02 23:37:29.342596 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:29.342469 5076 scope.go:122] "RemoveContainer" containerID="fa0b7f5ace5dd7309266db5f2ebd09b16fb0911167827e14bee90bc76bfae09c" Nov 02 23:37:29.344045 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:29.344017 5076 generic.go:356] "Generic (PLEG): container finished" podID="89a8789b-2ff7-4c73-9bee-2d3879cbe2b9" containerID="eaf11421bb0c333736917c51669b4f46a8c723acfc901c22d7529dd49bb22b7b" exitCode=0 Nov 02 23:37:29.344173 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:29.344060 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5ff336-7d447bd878-v7jmb" Nov 02 23:37:29.344319 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:29.344060 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5ff336-7d447bd878-v7jmb" event={"ID":"89a8789b-2ff7-4c73-9bee-2d3879cbe2b9","Type":"ContainerDied","Data":"eaf11421bb0c333736917c51669b4f46a8c723acfc901c22d7529dd49bb22b7b"} Nov 02 23:37:29.344457 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:29.344353 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5ff336-7d447bd878-v7jmb" event={"ID":"89a8789b-2ff7-4c73-9bee-2d3879cbe2b9","Type":"ContainerDied","Data":"ceb345e51032b5f34a95a65189fcde71014021c3cc69f6ec0ae0a93a867419e6"} Nov 02 23:37:29.352520 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:29.352501 5076 scope.go:122] "RemoveContainer" containerID="fa0b7f5ace5dd7309266db5f2ebd09b16fb0911167827e14bee90bc76bfae09c" Nov 02 23:37:29.352794 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:37:29.352771 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa0b7f5ace5dd7309266db5f2ebd09b16fb0911167827e14bee90bc76bfae09c\": container with ID starting with fa0b7f5ace5dd7309266db5f2ebd09b16fb0911167827e14bee90bc76bfae09c not found: ID does not exist" containerID="fa0b7f5ace5dd7309266db5f2ebd09b16fb0911167827e14bee90bc76bfae09c" Nov 02 23:37:29.352848 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:29.352801 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa0b7f5ace5dd7309266db5f2ebd09b16fb0911167827e14bee90bc76bfae09c"} err="failed to get container status \"fa0b7f5ace5dd7309266db5f2ebd09b16fb0911167827e14bee90bc76bfae09c\": rpc error: code = NotFound desc = could not find container \"fa0b7f5ace5dd7309266db5f2ebd09b16fb0911167827e14bee90bc76bfae09c\": container with ID starting with fa0b7f5ace5dd7309266db5f2ebd09b16fb0911167827e14bee90bc76bfae09c not found: ID does not exist" Nov 02 23:37:29.352848 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:29.352819 5076 scope.go:122] "RemoveContainer" containerID="eaf11421bb0c333736917c51669b4f46a8c723acfc901c22d7529dd49bb22b7b" Nov 02 23:37:29.361623 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:29.361445 5076 scope.go:122] "RemoveContainer" containerID="eaf11421bb0c333736917c51669b4f46a8c723acfc901c22d7529dd49bb22b7b" Nov 02 23:37:29.362107 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:37:29.361801 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaf11421bb0c333736917c51669b4f46a8c723acfc901c22d7529dd49bb22b7b\": container with ID starting with eaf11421bb0c333736917c51669b4f46a8c723acfc901c22d7529dd49bb22b7b not found: ID does not exist" containerID="eaf11421bb0c333736917c51669b4f46a8c723acfc901c22d7529dd49bb22b7b" Nov 02 23:37:29.362107 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:29.361836 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaf11421bb0c333736917c51669b4f46a8c723acfc901c22d7529dd49bb22b7b"} err="failed to get container status \"eaf11421bb0c333736917c51669b4f46a8c723acfc901c22d7529dd49bb22b7b\": rpc error: code = NotFound desc = could not find container \"eaf11421bb0c333736917c51669b4f46a8c723acfc901c22d7529dd49bb22b7b\": container with ID starting with eaf11421bb0c333736917c51669b4f46a8c723acfc901c22d7529dd49bb22b7b not found: ID does not exist" Nov 02 23:37:29.498782 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:29.498740 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-5fdb8d5c9f-8xdhn"] Nov 02 23:37:29.501388 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:29.501356 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4"] Nov 02 23:37:29.502129 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:29.502105 5076 cpu_manager.go:414] "containerMap: removing container" podUID="a6ef156b-92aa-4950-b83e-294e4adc2e77" containerName="payload-pre-processing" Nov 02 23:37:29.502266 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:29.502130 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6ef156b-92aa-4950-b83e-294e4adc2e77" containerName="payload-pre-processing" Nov 02 23:37:29.502266 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:29.502194 5076 cpu_manager.go:414] "containerMap: removing container" podUID="89a8789b-2ff7-4c73-9bee-2d3879cbe2b9" containerName="payload-processing" Nov 02 23:37:29.502266 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:29.502202 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89a8789b-2ff7-4c73-9bee-2d3879cbe2b9" containerName="payload-processing" Nov 02 23:37:29.502433 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:29.502327 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="89a8789b-2ff7-4c73-9bee-2d3879cbe2b9" containerName="payload-processing" Nov 02 23:37:29.502433 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:29.502343 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6ef156b-92aa-4950-b83e-294e4adc2e77" containerName="payload-pre-processing" Nov 02 23:37:29.506985 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:29.506965 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-5fdb8d5c9f-8xdhn"] Nov 02 23:37:29.507079 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:29.507052 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4" Nov 02 23:37:29.556729 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:29.556703 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-9672a1-serving-cert\"" Nov 02 23:37:29.558864 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:29.558836 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4"] Nov 02 23:37:29.600136 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:29.600106 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e316f95-0005-44bd-a11b-8b04b3e32297-maas-api-tls\") pod \"maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4\" (UID: \"5e316f95-0005-44bd-a11b-8b04b3e32297\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4" Nov 02 23:37:29.600298 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:29.600140 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e316f95-0005-44bd-a11b-8b04b3e32297-metrics-certs\") pod \"maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4\" (UID: \"5e316f95-0005-44bd-a11b-8b04b3e32297\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4" Nov 02 23:37:29.600298 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:29.600286 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq8dv\" (UniqueName: \"kubernetes.io/projected/5e316f95-0005-44bd-a11b-8b04b3e32297-kube-api-access-hq8dv\") pod \"maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4\" (UID: \"5e316f95-0005-44bd-a11b-8b04b3e32297\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4" Nov 02 23:37:29.600372 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:29.600357 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e316f95-0005-44bd-a11b-8b04b3e32297-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4\" (UID: \"5e316f95-0005-44bd-a11b-8b04b3e32297\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4" Nov 02 23:37:29.607798 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:29.607767 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5ff336-7d447bd878-v7jmb"] Nov 02 23:37:29.611741 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:29.611717 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5ff336-7d447bd878-v7jmb"] Nov 02 23:37:29.701092 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:29.701057 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e316f95-0005-44bd-a11b-8b04b3e32297-maas-api-tls\") pod \"maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4\" (UID: \"5e316f95-0005-44bd-a11b-8b04b3e32297\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4" Nov 02 23:37:29.701092 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:29.701090 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e316f95-0005-44bd-a11b-8b04b3e32297-metrics-certs\") pod \"maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4\" (UID: \"5e316f95-0005-44bd-a11b-8b04b3e32297\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4" Nov 02 23:37:29.701337 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:29.701171 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq8dv\" (UniqueName: \"kubernetes.io/projected/5e316f95-0005-44bd-a11b-8b04b3e32297-kube-api-access-hq8dv\") pod \"maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4\" (UID: \"5e316f95-0005-44bd-a11b-8b04b3e32297\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4" Nov 02 23:37:29.701337 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:37:29.701212 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-9672a1-serving-cert: secret "maas-api-e2e-praxis-a-9672a1-serving-cert" not found Nov 02 23:37:29.701337 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:29.701235 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e316f95-0005-44bd-a11b-8b04b3e32297-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4\" (UID: \"5e316f95-0005-44bd-a11b-8b04b3e32297\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4" Nov 02 23:37:29.701337 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:37:29.701277 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e316f95-0005-44bd-a11b-8b04b3e32297-maas-api-tls podName:5e316f95-0005-44bd-a11b-8b04b3e32297 nodeName:}" failed. No retries permitted until 2026-11-02 23:37:30.201257827 +0000 UTC m=+2699.169789609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5e316f95-0005-44bd-a11b-8b04b3e32297-maas-api-tls") pod "maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4" (UID: "5e316f95-0005-44bd-a11b-8b04b3e32297") : secret "maas-api-e2e-praxis-a-9672a1-serving-cert" not found Nov 02 23:37:29.701842 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:29.701821 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e316f95-0005-44bd-a11b-8b04b3e32297-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4\" (UID: \"5e316f95-0005-44bd-a11b-8b04b3e32297\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4" Nov 02 23:37:29.703523 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:29.703503 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e316f95-0005-44bd-a11b-8b04b3e32297-metrics-certs\") pod \"maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4\" (UID: \"5e316f95-0005-44bd-a11b-8b04b3e32297\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4" Nov 02 23:37:29.781959 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:29.772868 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq8dv\" (UniqueName: \"kubernetes.io/projected/5e316f95-0005-44bd-a11b-8b04b3e32297-kube-api-access-hq8dv\") pod \"maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4\" (UID: \"5e316f95-0005-44bd-a11b-8b04b3e32297\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4" Nov 02 23:37:30.208518 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:30.208482 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e316f95-0005-44bd-a11b-8b04b3e32297-maas-api-tls\") pod \"maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4\" (UID: \"5e316f95-0005-44bd-a11b-8b04b3e32297\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4" Nov 02 23:37:30.211010 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:30.210985 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e316f95-0005-44bd-a11b-8b04b3e32297-maas-api-tls\") pod \"maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4\" (UID: \"5e316f95-0005-44bd-a11b-8b04b3e32297\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4" Nov 02 23:37:30.421045 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:30.421007 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4" Nov 02 23:37:30.790863 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:30.790833 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4"] Nov 02 23:37:30.792126 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:37:30.792094 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e316f95_0005_44bd_a11b_8b04b3e32297.slice/crio-f2724fc1e872aec001ad2ff5b204f5e6919a9704dafda1a48a64a743f7eb38c6 WatchSource:0}: Error finding container f2724fc1e872aec001ad2ff5b204f5e6919a9704dafda1a48a64a743f7eb38c6: Status 404 returned error can't find the container with id f2724fc1e872aec001ad2ff5b204f5e6919a9704dafda1a48a64a743f7eb38c6 Nov 02 23:37:30.793608 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:30.793589 5076 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 23:37:31.148478 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:31.148391 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89a8789b-2ff7-4c73-9bee-2d3879cbe2b9" path="/var/lib/kubelet/pods/89a8789b-2ff7-4c73-9bee-2d3879cbe2b9/volumes" Nov 02 23:37:31.148829 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:31.148814 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6ef156b-92aa-4950-b83e-294e4adc2e77" path="/var/lib/kubelet/pods/a6ef156b-92aa-4950-b83e-294e4adc2e77/volumes" Nov 02 23:37:31.360946 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:31.360909 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4" event={"ID":"5e316f95-0005-44bd-a11b-8b04b3e32297","Type":"ContainerStarted","Data":"f2724fc1e872aec001ad2ff5b204f5e6919a9704dafda1a48a64a743f7eb38c6"} Nov 02 23:37:31.845964 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:31.845941 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 02 23:37:31.877762 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:31.877739 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 02 23:37:32.370949 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:32.370914 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4" event={"ID":"5e316f95-0005-44bd-a11b-8b04b3e32297","Type":"ContainerStarted","Data":"ca42ef19bab30ca0be9870bf41c21b8ace331ebeb55ecfca5db6c87a26b37dc6"} Nov 02 23:37:32.371120 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:32.370980 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4" Nov 02 23:37:32.496949 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:32.494014 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4" podStartSLOduration=2.476879508 podStartE2EDuration="3.493995054s" totalImagesPullingTime="1.017115546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:37:32.49235388 +0000 UTC m=+2701.460885664" watchObservedRunningTime="2026-11-02 23:37:32.493995054 +0000 UTC m=+2701.462526839" Nov 02 23:37:34.447844 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:34.447805 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv"] Nov 02 23:37:34.448363 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:34.448095 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" podUID="ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9" containerName="istio-proxy" containerID="cri-o://b76d3473da76180aac00b85fba07f1352d9e22291b807744217a387f1d564641" gracePeriod=30 Nov 02 23:37:38.382804 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:38.382768 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4" Nov 02 23:37:39.669390 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:39.669356 5076 generic.go:356] "Generic (PLEG): container finished" podID="ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9" containerID="b76d3473da76180aac00b85fba07f1352d9e22291b807744217a387f1d564641" exitCode=0 Nov 02 23:37:39.669867 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:39.669396 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" event={"ID":"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9","Type":"ContainerDied","Data":"b76d3473da76180aac00b85fba07f1352d9e22291b807744217a387f1d564641"} Nov 02 23:37:39.669867 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:39.669461 5076 scope.go:122] "RemoveContainer" containerID="25d296194a17393f8f3e7d15b994b96d25cbb18c18540c4c94e0ecca5d3da43a" Nov 02 23:37:39.743859 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:39.743829 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" Nov 02 23:37:39.747996 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:39.747967 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5"] Nov 02 23:37:39.748631 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:39.748606 5076 cpu_manager.go:414] "containerMap: removing container" podUID="ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9" containerName="istio-proxy" Nov 02 23:37:39.748631 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:39.748631 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9" containerName="istio-proxy" Nov 02 23:37:39.748794 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:39.748774 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9" containerName="istio-proxy" Nov 02 23:37:39.748794 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:39.748791 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9" containerName="istio-proxy" Nov 02 23:37:39.752117 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:39.752098 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" Nov 02 23:37:39.837474 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:39.837432 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5"] Nov 02 23:37:39.860851 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:39.860806 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-c5148c-openshift-default-dockercfg-j4nxl\"" Nov 02 23:37:39.941431 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:37:39.941387 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:37:39.979272 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:39.975809 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-credential-socket\") pod \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\" (UID: \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\") " Nov 02 23:37:39.979272 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:39.975868 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-workload-certs\") pod \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\" (UID: \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\") " Nov 02 23:37:39.979272 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:39.975914 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-workload-socket\") pod \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\" (UID: \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\") " Nov 02 23:37:39.979272 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:39.975993 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-istiod-ca-cert\") pod \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\" (UID: \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\") " Nov 02 23:37:39.979272 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:39.976090 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-istio-podinfo\") pod \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\" (UID: \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\") " Nov 02 23:37:39.979272 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:39.976131 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-istio-token\" (UniqueName: \"kubernetes.io/projected/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-istio-token\") pod \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\" (UID: \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\") " Nov 02 23:37:39.979272 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:39.976164 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-kube-api-access-vpvzk\" (UniqueName: \"kubernetes.io/projected/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-kube-api-access-vpvzk\") pod \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\" (UID: \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\") " Nov 02 23:37:39.979272 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:39.976208 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-istio-data\") pod \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\" (UID: \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\") " Nov 02 23:37:39.979272 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:39.976274 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-istio-envoy\") pod \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\" (UID: \"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9\") " Nov 02 23:37:39.979272 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:39.976515 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51ba5367-449d-4292-9868-6ff46c0f94d3-istio-data\") pod \"e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5\" (UID: \"51ba5367-449d-4292-9868-6ff46c0f94d3\") " pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" Nov 02 23:37:39.979272 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:39.976572 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ba5367-449d-4292-9868-6ff46c0f94d3-workload-socket\") pod \"e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5\" (UID: \"51ba5367-449d-4292-9868-6ff46c0f94d3\") " pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" Nov 02 23:37:39.979272 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:39.976653 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd9ln\" (UniqueName: \"kubernetes.io/projected/51ba5367-449d-4292-9868-6ff46c0f94d3-kube-api-access-fd9ln\") pod \"e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5\" (UID: \"51ba5367-449d-4292-9868-6ff46c0f94d3\") " pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" Nov 02 23:37:39.979272 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:39.976729 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51ba5367-449d-4292-9868-6ff46c0f94d3-istio-token\") pod \"e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5\" (UID: \"51ba5367-449d-4292-9868-6ff46c0f94d3\") " pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" Nov 02 23:37:39.979272 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:39.976808 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51ba5367-449d-4292-9868-6ff46c0f94d3-istiod-ca-cert\") pod \"e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5\" (UID: \"51ba5367-449d-4292-9868-6ff46c0f94d3\") " pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" Nov 02 23:37:39.979272 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:39.976839 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51ba5367-449d-4292-9868-6ff46c0f94d3-workload-certs\") pod \"e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5\" (UID: \"51ba5367-449d-4292-9868-6ff46c0f94d3\") " pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" Nov 02 23:37:39.979272 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:39.976863 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ba5367-449d-4292-9868-6ff46c0f94d3-credential-socket\") pod \"e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5\" (UID: \"51ba5367-449d-4292-9868-6ff46c0f94d3\") " pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" Nov 02 23:37:39.979272 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:39.976905 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51ba5367-449d-4292-9868-6ff46c0f94d3-istio-envoy\") pod \"e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5\" (UID: \"51ba5367-449d-4292-9868-6ff46c0f94d3\") " pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" Nov 02 23:37:39.979272 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:39.976954 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51ba5367-449d-4292-9868-6ff46c0f94d3-istio-podinfo\") pod \"e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5\" (UID: \"51ba5367-449d-4292-9868-6ff46c0f94d3\") " pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" Nov 02 23:37:40.012758 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:39.994295 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-workload-socket" pod "ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9" (UID: "ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:37:40.012758 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:39.994500 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-workload-certs" pod "ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9" (UID: "ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:37:40.012758 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.000669 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-istiod-ca-cert" pod "ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9" (UID: "ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:37:40.012758 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.001082 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-credential-socket" pod "ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9" (UID: "ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:37:40.012758 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.002770 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-istio-data" pod "ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9" (UID: "ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:37:40.012758 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.003586 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-kube-api-access-vpvzk" pod "ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9" (UID: "ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9"). InnerVolumeSpecName "kube-api-access-vpvzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:37:40.018645 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.018162 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-istio-envoy" pod "ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9" (UID: "ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:37:40.018645 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.018412 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-istio-token" pod "ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9" (UID: "ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:37:40.018937 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.018532 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-istio-podinfo" pod "ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9" (UID: "ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:37:40.078314 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.078274 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51ba5367-449d-4292-9868-6ff46c0f94d3-istio-token\") pod \"e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5\" (UID: \"51ba5367-449d-4292-9868-6ff46c0f94d3\") " pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" Nov 02 23:37:40.078536 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.078341 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51ba5367-449d-4292-9868-6ff46c0f94d3-istiod-ca-cert\") pod \"e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5\" (UID: \"51ba5367-449d-4292-9868-6ff46c0f94d3\") " pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" Nov 02 23:37:40.078536 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.078449 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51ba5367-449d-4292-9868-6ff46c0f94d3-workload-certs\") pod \"e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5\" (UID: \"51ba5367-449d-4292-9868-6ff46c0f94d3\") " pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" Nov 02 23:37:40.078536 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.078481 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ba5367-449d-4292-9868-6ff46c0f94d3-credential-socket\") pod \"e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5\" (UID: \"51ba5367-449d-4292-9868-6ff46c0f94d3\") " pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" Nov 02 23:37:40.078536 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.078518 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51ba5367-449d-4292-9868-6ff46c0f94d3-istio-envoy\") pod \"e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5\" (UID: \"51ba5367-449d-4292-9868-6ff46c0f94d3\") " pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" Nov 02 23:37:40.078781 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.078567 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51ba5367-449d-4292-9868-6ff46c0f94d3-istio-podinfo\") pod \"e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5\" (UID: \"51ba5367-449d-4292-9868-6ff46c0f94d3\") " pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" Nov 02 23:37:40.078781 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.078621 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51ba5367-449d-4292-9868-6ff46c0f94d3-istio-data\") pod \"e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5\" (UID: \"51ba5367-449d-4292-9868-6ff46c0f94d3\") " pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" Nov 02 23:37:40.078781 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.078654 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ba5367-449d-4292-9868-6ff46c0f94d3-workload-socket\") pod \"e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5\" (UID: \"51ba5367-449d-4292-9868-6ff46c0f94d3\") " pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" Nov 02 23:37:40.078781 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.078733 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd9ln\" (UniqueName: \"kubernetes.io/projected/51ba5367-449d-4292-9868-6ff46c0f94d3-kube-api-access-fd9ln\") pod \"e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5\" (UID: \"51ba5367-449d-4292-9868-6ff46c0f94d3\") " pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" Nov 02 23:37:40.079217 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.078851 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:40.079217 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.078854 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ba5367-449d-4292-9868-6ff46c0f94d3-credential-socket\") pod \"e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5\" (UID: \"51ba5367-449d-4292-9868-6ff46c0f94d3\") " pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" Nov 02 23:37:40.079217 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.078870 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:40.079217 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.078904 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:40.079217 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.078917 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpvzk\" (UniqueName: \"kubernetes.io/projected/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-kube-api-access-vpvzk\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:40.079217 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.078930 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:40.079217 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.078946 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:40.079217 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.078966 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:40.079217 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.078982 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:40.079217 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.079022 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:40.079217 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.078964 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51ba5367-449d-4292-9868-6ff46c0f94d3-workload-certs\") pod \"e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5\" (UID: \"51ba5367-449d-4292-9868-6ff46c0f94d3\") " pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" Nov 02 23:37:40.079217 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.079151 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ba5367-449d-4292-9868-6ff46c0f94d3-workload-socket\") pod \"e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5\" (UID: \"51ba5367-449d-4292-9868-6ff46c0f94d3\") " pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" Nov 02 23:37:40.079217 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.079216 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51ba5367-449d-4292-9868-6ff46c0f94d3-istio-data\") pod \"e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5\" (UID: \"51ba5367-449d-4292-9868-6ff46c0f94d3\") " pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" Nov 02 23:37:40.079852 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.079493 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51ba5367-449d-4292-9868-6ff46c0f94d3-istiod-ca-cert\") pod \"e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5\" (UID: \"51ba5367-449d-4292-9868-6ff46c0f94d3\") " pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" Nov 02 23:37:40.080953 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.080908 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51ba5367-449d-4292-9868-6ff46c0f94d3-istio-envoy\") pod \"e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5\" (UID: \"51ba5367-449d-4292-9868-6ff46c0f94d3\") " pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" Nov 02 23:37:40.081171 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.081152 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51ba5367-449d-4292-9868-6ff46c0f94d3-istio-podinfo\") pod \"e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5\" (UID: \"51ba5367-449d-4292-9868-6ff46c0f94d3\") " pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" Nov 02 23:37:40.143733 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.136103 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd9ln\" (UniqueName: \"kubernetes.io/projected/51ba5367-449d-4292-9868-6ff46c0f94d3-kube-api-access-fd9ln\") pod \"e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5\" (UID: \"51ba5367-449d-4292-9868-6ff46c0f94d3\") " pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" Nov 02 23:37:40.143733 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.136613 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51ba5367-449d-4292-9868-6ff46c0f94d3-istio-token\") pod \"e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5\" (UID: \"51ba5367-449d-4292-9868-6ff46c0f94d3\") " pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" Nov 02 23:37:40.365619 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.365525 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" Nov 02 23:37:40.677642 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.677573 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" Nov 02 23:37:40.677642 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.677618 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv" event={"ID":"ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9","Type":"ContainerDied","Data":"d63e2b20e38dfbd3b9557a99a59b1468f14b8e4f4f116469d51ffa091bc888f3"} Nov 02 23:37:40.678153 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.677674 5076 scope.go:122] "RemoveContainer" containerID="b76d3473da76180aac00b85fba07f1352d9e22291b807744217a387f1d564641" Nov 02 23:37:40.782710 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.782686 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5"] Nov 02 23:37:40.784066 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:37:40.784039 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51ba5367_449d_4292_9868_6ff46c0f94d3.slice/crio-b71d144c4443c05eed648d0b63dfc647a0ec440c0490314be19d387c4d586f7a WatchSource:0}: Error finding container b71d144c4443c05eed648d0b63dfc647a0ec440c0490314be19d387c4d586f7a: Status 404 returned error can't find the container with id b71d144c4443c05eed648d0b63dfc647a0ec440c0490314be19d387c4d586f7a Nov 02 23:37:40.786412 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.786383 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:37:40.786508 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.786444 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:37:40.786508 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.786475 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:37:40.862536 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.849073 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv"] Nov 02 23:37:40.862536 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:40.850815 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-5ff336-openshift-default-68fcfbbc79-26dsv"] Nov 02 23:37:41.150499 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:41.150460 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9" path="/var/lib/kubelet/pods/ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9/volumes" Nov 02 23:37:41.685371 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:41.685333 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" event={"ID":"51ba5367-449d-4292-9868-6ff46c0f94d3","Type":"ContainerStarted","Data":"7c99d708208871f9cca3b3e1f9608110d458c6fd09204a5433513332f9d7238f"} Nov 02 23:37:41.685850 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:41.685397 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" event={"ID":"51ba5367-449d-4292-9868-6ff46c0f94d3","Type":"ContainerStarted","Data":"b71d144c4443c05eed648d0b63dfc647a0ec440c0490314be19d387c4d586f7a"} Nov 02 23:37:41.797409 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:41.797335 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" podStartSLOduration=2.797315479 podStartE2EDuration="2.797315479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:37:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:37:41.79662458 +0000 UTC m=+2710.765156364" watchObservedRunningTime="2026-11-02 23:37:41.797315479 +0000 UTC m=+2710.765847264" Nov 02 23:37:42.365818 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:42.365778 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" Nov 02 23:37:42.371472 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:42.371447 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" Nov 02 23:37:42.691782 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:42.691756 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" Nov 02 23:37:42.692843 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:42.692824 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" Nov 02 23:37:53.511332 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:53.511286 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr"] Nov 02 23:37:53.512107 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:53.512084 5076 cpu_manager.go:414] "containerMap: removing container" podUID="ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9" containerName="istio-proxy" Nov 02 23:37:53.512241 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:53.512108 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef0ec1d5-f3b2-4b9e-a86b-d2a158f997c9" containerName="istio-proxy" Nov 02 23:37:53.516904 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:53.516870 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" Nov 02 23:37:53.574915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:53.568974 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-fe63ac-openshift-default-dockercfg-27j7l\"" Nov 02 23:37:53.595913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:53.592586 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4deff109-12f0-42e8-8908-508e7f6eec68-credential-socket\") pod \"e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr\" (UID: \"4deff109-12f0-42e8-8908-508e7f6eec68\") " pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" Nov 02 23:37:53.595913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:53.592642 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4deff109-12f0-42e8-8908-508e7f6eec68-workload-socket\") pod \"e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr\" (UID: \"4deff109-12f0-42e8-8908-508e7f6eec68\") " pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" Nov 02 23:37:53.595913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:53.592705 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4deff109-12f0-42e8-8908-508e7f6eec68-workload-certs\") pod \"e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr\" (UID: \"4deff109-12f0-42e8-8908-508e7f6eec68\") " pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" Nov 02 23:37:53.595913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:53.592758 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgnpt\" (UniqueName: \"kubernetes.io/projected/4deff109-12f0-42e8-8908-508e7f6eec68-kube-api-access-tgnpt\") pod \"e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr\" (UID: \"4deff109-12f0-42e8-8908-508e7f6eec68\") " pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" Nov 02 23:37:53.595913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:53.592804 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4deff109-12f0-42e8-8908-508e7f6eec68-istio-podinfo\") pod \"e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr\" (UID: \"4deff109-12f0-42e8-8908-508e7f6eec68\") " pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" Nov 02 23:37:53.595913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:53.592906 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4deff109-12f0-42e8-8908-508e7f6eec68-istio-data\") pod \"e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr\" (UID: \"4deff109-12f0-42e8-8908-508e7f6eec68\") " pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" Nov 02 23:37:53.595913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:53.592954 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4deff109-12f0-42e8-8908-508e7f6eec68-istio-token\") pod \"e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr\" (UID: \"4deff109-12f0-42e8-8908-508e7f6eec68\") " pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" Nov 02 23:37:53.595913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:53.592985 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4deff109-12f0-42e8-8908-508e7f6eec68-istiod-ca-cert\") pod \"e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr\" (UID: \"4deff109-12f0-42e8-8908-508e7f6eec68\") " pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" Nov 02 23:37:53.595913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:53.593068 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4deff109-12f0-42e8-8908-508e7f6eec68-istio-envoy\") pod \"e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr\" (UID: \"4deff109-12f0-42e8-8908-508e7f6eec68\") " pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" Nov 02 23:37:53.620170 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:53.620063 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr"] Nov 02 23:37:53.694706 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:53.694660 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4deff109-12f0-42e8-8908-508e7f6eec68-credential-socket\") pod \"e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr\" (UID: \"4deff109-12f0-42e8-8908-508e7f6eec68\") " pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" Nov 02 23:37:53.694706 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:53.694710 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4deff109-12f0-42e8-8908-508e7f6eec68-workload-socket\") pod \"e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr\" (UID: \"4deff109-12f0-42e8-8908-508e7f6eec68\") " pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" Nov 02 23:37:53.694996 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:53.694761 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4deff109-12f0-42e8-8908-508e7f6eec68-workload-certs\") pod \"e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr\" (UID: \"4deff109-12f0-42e8-8908-508e7f6eec68\") " pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" Nov 02 23:37:53.694996 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:53.694797 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgnpt\" (UniqueName: \"kubernetes.io/projected/4deff109-12f0-42e8-8908-508e7f6eec68-kube-api-access-tgnpt\") pod \"e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr\" (UID: \"4deff109-12f0-42e8-8908-508e7f6eec68\") " pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" Nov 02 23:37:53.694996 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:53.694839 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4deff109-12f0-42e8-8908-508e7f6eec68-istio-podinfo\") pod \"e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr\" (UID: \"4deff109-12f0-42e8-8908-508e7f6eec68\") " pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" Nov 02 23:37:53.694996 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:53.694942 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4deff109-12f0-42e8-8908-508e7f6eec68-istio-data\") pod \"e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr\" (UID: \"4deff109-12f0-42e8-8908-508e7f6eec68\") " pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" Nov 02 23:37:53.695203 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:53.695052 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4deff109-12f0-42e8-8908-508e7f6eec68-istio-token\") pod \"e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr\" (UID: \"4deff109-12f0-42e8-8908-508e7f6eec68\") " pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" Nov 02 23:37:53.695203 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:53.695089 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4deff109-12f0-42e8-8908-508e7f6eec68-istiod-ca-cert\") pod \"e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr\" (UID: \"4deff109-12f0-42e8-8908-508e7f6eec68\") " pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" Nov 02 23:37:53.695203 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:53.695114 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4deff109-12f0-42e8-8908-508e7f6eec68-credential-socket\") pod \"e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr\" (UID: \"4deff109-12f0-42e8-8908-508e7f6eec68\") " pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" Nov 02 23:37:53.695364 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:53.695218 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4deff109-12f0-42e8-8908-508e7f6eec68-istio-envoy\") pod \"e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr\" (UID: \"4deff109-12f0-42e8-8908-508e7f6eec68\") " pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" Nov 02 23:37:53.695364 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:53.695311 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4deff109-12f0-42e8-8908-508e7f6eec68-workload-certs\") pod \"e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr\" (UID: \"4deff109-12f0-42e8-8908-508e7f6eec68\") " pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" Nov 02 23:37:53.695364 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:53.695345 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4deff109-12f0-42e8-8908-508e7f6eec68-workload-socket\") pod \"e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr\" (UID: \"4deff109-12f0-42e8-8908-508e7f6eec68\") " pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" Nov 02 23:37:53.695521 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:53.695476 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4deff109-12f0-42e8-8908-508e7f6eec68-istio-data\") pod \"e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr\" (UID: \"4deff109-12f0-42e8-8908-508e7f6eec68\") " pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" Nov 02 23:37:53.695981 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:53.695961 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4deff109-12f0-42e8-8908-508e7f6eec68-istiod-ca-cert\") pod \"e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr\" (UID: \"4deff109-12f0-42e8-8908-508e7f6eec68\") " pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" Nov 02 23:37:53.697444 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:53.697420 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4deff109-12f0-42e8-8908-508e7f6eec68-istio-envoy\") pod \"e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr\" (UID: \"4deff109-12f0-42e8-8908-508e7f6eec68\") " pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" Nov 02 23:37:53.697637 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:53.697619 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4deff109-12f0-42e8-8908-508e7f6eec68-istio-podinfo\") pod \"e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr\" (UID: \"4deff109-12f0-42e8-8908-508e7f6eec68\") " pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" Nov 02 23:37:53.756697 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:53.756521 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgnpt\" (UniqueName: \"kubernetes.io/projected/4deff109-12f0-42e8-8908-508e7f6eec68-kube-api-access-tgnpt\") pod \"e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr\" (UID: \"4deff109-12f0-42e8-8908-508e7f6eec68\") " pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" Nov 02 23:37:53.768722 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:53.768647 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4deff109-12f0-42e8-8908-508e7f6eec68-istio-token\") pod \"e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr\" (UID: \"4deff109-12f0-42e8-8908-508e7f6eec68\") " pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" Nov 02 23:37:53.831294 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:53.831258 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" Nov 02 23:37:54.250866 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:54.250837 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr"] Nov 02 23:37:54.251089 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:37:54.251063 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4deff109_12f0_42e8_8908_508e7f6eec68.slice/crio-9aa2c3e29b72bf9f9b6c67ce3545f5f322540425745f742d0166f0ad53d20566 WatchSource:0}: Error finding container 9aa2c3e29b72bf9f9b6c67ce3545f5f322540425745f742d0166f0ad53d20566: Status 404 returned error can't find the container with id 9aa2c3e29b72bf9f9b6c67ce3545f5f322540425745f742d0166f0ad53d20566 Nov 02 23:37:54.253709 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:54.253681 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:37:54.253800 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:54.253742 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:37:54.253800 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:54.253772 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:37:54.773783 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:54.773738 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" event={"ID":"4deff109-12f0-42e8-8908-508e7f6eec68","Type":"ContainerStarted","Data":"e64290c914e0b51b805d0321fa2662f5f72e4e7b44f8e746614271b05e4e5b50"} Nov 02 23:37:54.774216 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:54.773823 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" event={"ID":"4deff109-12f0-42e8-8908-508e7f6eec68","Type":"ContainerStarted","Data":"9aa2c3e29b72bf9f9b6c67ce3545f5f322540425745f742d0166f0ad53d20566"} Nov 02 23:37:54.831457 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:54.831425 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" Nov 02 23:37:54.886362 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:54.886295 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" podStartSLOduration=1.886279772 podStartE2EDuration="1.886279772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:37:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:37:54.883626059 +0000 UTC m=+2723.852157844" watchObservedRunningTime="2026-11-02 23:37:54.886279772 +0000 UTC m=+2723.854811555" Nov 02 23:37:55.838680 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:55.838650 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" Nov 02 23:37:56.126917 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:56.126822 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw"] Nov 02 23:37:56.131065 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:56.131042 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" Nov 02 23:37:56.187811 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:56.187773 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw"] Nov 02 23:37:56.227061 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:56.227011 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-credential-socket\") pod \"e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw\" (UID: \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\") " pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" Nov 02 23:37:56.227253 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:56.227067 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-workload-certs\") pod \"e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw\" (UID: \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\") " pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" Nov 02 23:37:56.227253 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:56.227144 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-istio-token\") pod \"e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw\" (UID: \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\") " pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" Nov 02 23:37:56.227348 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:56.227257 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-istio-envoy\") pod \"e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw\" (UID: \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\") " pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" Nov 02 23:37:56.227348 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:56.227321 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-istiod-ca-cert\") pod \"e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw\" (UID: \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\") " pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" Nov 02 23:37:56.227442 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:56.227424 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-istio-data\") pod \"e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw\" (UID: \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\") " pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" Nov 02 23:37:56.227481 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:56.227465 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-istio-podinfo\") pod \"e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw\" (UID: \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\") " pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" Nov 02 23:37:56.227615 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:56.227593 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cg4t\" (UniqueName: \"kubernetes.io/projected/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-kube-api-access-9cg4t\") pod \"e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw\" (UID: \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\") " pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" Nov 02 23:37:56.227701 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:56.227664 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-workload-socket\") pod \"e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw\" (UID: \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\") " pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" Nov 02 23:37:56.260320 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:56.260287 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-b4d53e-openshift-default-dockercfg-2fvhd\"" Nov 02 23:37:56.328648 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:56.328605 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-credential-socket\") pod \"e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw\" (UID: \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\") " pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" Nov 02 23:37:56.343966 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:56.329397 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-workload-certs\") pod \"e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw\" (UID: \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\") " pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" Nov 02 23:37:56.345686 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:56.345643 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-istio-token\") pod \"e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw\" (UID: \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\") " pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" Nov 02 23:37:56.345980 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:56.345958 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-istio-envoy\") pod \"e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw\" (UID: \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\") " pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" Nov 02 23:37:56.346174 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:56.346148 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-istiod-ca-cert\") pod \"e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw\" (UID: \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\") " pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" Nov 02 23:37:56.346337 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:56.346313 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-istio-data\") pod \"e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw\" (UID: \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\") " pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" Nov 02 23:37:56.346492 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:56.346468 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-istio-podinfo\") pod \"e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw\" (UID: \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\") " pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" Nov 02 23:37:56.346650 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:56.346636 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cg4t\" (UniqueName: \"kubernetes.io/projected/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-kube-api-access-9cg4t\") pod \"e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw\" (UID: \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\") " pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" Nov 02 23:37:56.346804 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:56.346781 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-workload-socket\") pod \"e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw\" (UID: \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\") " pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" Nov 02 23:37:56.347450 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:56.347412 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-workload-socket\") pod \"e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw\" (UID: \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\") " pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" Nov 02 23:37:56.348681 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:56.334377 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-workload-certs\") pod \"e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw\" (UID: \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\") " pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" Nov 02 23:37:56.348820 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:56.348141 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-istio-data\") pod \"e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw\" (UID: \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\") " pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" Nov 02 23:37:56.348952 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:56.329342 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-credential-socket\") pod \"e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw\" (UID: \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\") " pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" Nov 02 23:37:56.349854 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:56.349834 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-istiod-ca-cert\") pod \"e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw\" (UID: \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\") " pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" Nov 02 23:37:56.384970 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:56.376454 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-istio-envoy\") pod \"e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw\" (UID: \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\") " pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" Nov 02 23:37:56.396911 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:56.390458 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-istio-podinfo\") pod \"e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw\" (UID: \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\") " pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" Nov 02 23:37:56.426635 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:56.426599 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cg4t\" (UniqueName: \"kubernetes.io/projected/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-kube-api-access-9cg4t\") pod \"e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw\" (UID: \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\") " pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" Nov 02 23:37:56.434350 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:56.434317 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-istio-token\") pod \"e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw\" (UID: \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\") " pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" Nov 02 23:37:56.444833 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:56.444794 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" Nov 02 23:37:56.786170 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:56.786135 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" Nov 02 23:37:56.787255 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:56.787232 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" Nov 02 23:37:56.864943 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:37:56.864859 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba4e9d8b_de8c_49e2_a9a1_21f52d7747d2.slice/crio-a56b500f3b8cbca10f42c50090faeb6d4308bcc6bb3cc43d8da99f9dacfbd67a WatchSource:0}: Error finding container a56b500f3b8cbca10f42c50090faeb6d4308bcc6bb3cc43d8da99f9dacfbd67a: Status 404 returned error can't find the container with id a56b500f3b8cbca10f42c50090faeb6d4308bcc6bb3cc43d8da99f9dacfbd67a Nov 02 23:37:56.864943 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:56.864935 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw"] Nov 02 23:37:56.867566 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:56.867532 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:37:56.867628 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:56.867603 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:37:56.867671 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:56.867633 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:37:57.794137 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:57.794091 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" event={"ID":"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2","Type":"ContainerStarted","Data":"574dbc24446700e0bb5156ce4e1418b3041c05c7110bd1214dcaf87b8e76a31e"} Nov 02 23:37:57.794332 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:57.794158 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" event={"ID":"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2","Type":"ContainerStarted","Data":"a56b500f3b8cbca10f42c50090faeb6d4308bcc6bb3cc43d8da99f9dacfbd67a"} Nov 02 23:37:57.908730 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:57.908676 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" podStartSLOduration=1.908661261 podStartE2EDuration="1.908661261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:37:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:37:57.90724175 +0000 UTC m=+2726.875773535" watchObservedRunningTime="2026-11-02 23:37:57.908661261 +0000 UTC m=+2726.877193116" Nov 02 23:37:58.445718 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:58.445679 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" Nov 02 23:37:58.452832 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:58.452803 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" Nov 02 23:37:58.847954 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:58.847816 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" Nov 02 23:37:58.852618 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:37:58.852583 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" Nov 02 23:38:06.458856 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:06.458814 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc"] Nov 02 23:38:06.464619 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:06.464596 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" Nov 02 23:38:06.527070 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:06.527034 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-3d8404-openshift-default-dockercfg-kh456\"" Nov 02 23:38:06.576771 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:06.576731 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc"] Nov 02 23:38:06.587486 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:06.587454 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-istiod-ca-cert\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc\" (UID: \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" Nov 02 23:38:06.587858 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:06.587489 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-workload-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc\" (UID: \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" Nov 02 23:38:06.587858 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:06.587525 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-istio-data\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc\" (UID: \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" Nov 02 23:38:06.587858 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:06.587715 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-istio-envoy\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc\" (UID: \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" Nov 02 23:38:06.587858 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:06.587760 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-istio-token\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc\" (UID: \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" Nov 02 23:38:06.588083 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:06.587921 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-istio-podinfo\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc\" (UID: \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" Nov 02 23:38:06.588083 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:06.587961 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-workload-certs\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc\" (UID: \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" Nov 02 23:38:06.588083 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:06.588017 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-credential-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc\" (UID: \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" Nov 02 23:38:06.588083 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:06.588068 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v2bg\" (UniqueName: \"kubernetes.io/projected/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-kube-api-access-5v2bg\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc\" (UID: \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" Nov 02 23:38:06.688862 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:06.688825 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-istio-token\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc\" (UID: \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" Nov 02 23:38:06.689099 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:06.688919 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-istio-podinfo\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc\" (UID: \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" Nov 02 23:38:06.689099 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:06.688950 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-workload-certs\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc\" (UID: \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" Nov 02 23:38:06.689099 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:06.688985 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-credential-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc\" (UID: \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" Nov 02 23:38:06.689099 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:06.689013 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v2bg\" (UniqueName: \"kubernetes.io/projected/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-kube-api-access-5v2bg\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc\" (UID: \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" Nov 02 23:38:06.689099 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:06.689063 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-istiod-ca-cert\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc\" (UID: \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" Nov 02 23:38:06.689099 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:06.689078 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-workload-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc\" (UID: \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" Nov 02 23:38:06.689428 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:06.689118 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-istio-data\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc\" (UID: \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" Nov 02 23:38:06.689428 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:06.689215 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-istio-envoy\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc\" (UID: \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" Nov 02 23:38:06.689563 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:06.689531 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-workload-certs\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc\" (UID: \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" Nov 02 23:38:06.689563 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:06.689553 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-credential-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc\" (UID: \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" Nov 02 23:38:06.689659 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:06.689624 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-workload-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc\" (UID: \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" Nov 02 23:38:06.689695 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:06.689671 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-istio-data\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc\" (UID: \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" Nov 02 23:38:06.689990 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:06.689962 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-istiod-ca-cert\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc\" (UID: \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" Nov 02 23:38:06.691509 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:06.691481 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-istio-envoy\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc\" (UID: \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" Nov 02 23:38:06.691728 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:06.691710 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-istio-podinfo\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc\" (UID: \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" Nov 02 23:38:06.744227 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:06.744156 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-istio-token\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc\" (UID: \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" Nov 02 23:38:06.744227 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:06.744217 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v2bg\" (UniqueName: \"kubernetes.io/projected/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-kube-api-access-5v2bg\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc\" (UID: \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" Nov 02 23:38:06.779960 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:06.779927 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" Nov 02 23:38:07.228207 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:07.228179 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc"] Nov 02 23:38:07.229876 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:38:07.229847 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18e99da1_94f3_4d3e_a6ce_ee5a43219ce6.slice/crio-d3f1ec32bfb500659cc7c02d933d9ca0a0473715ad2a399cd3c86ad76a698af9 WatchSource:0}: Error finding container d3f1ec32bfb500659cc7c02d933d9ca0a0473715ad2a399cd3c86ad76a698af9: Status 404 returned error can't find the container with id d3f1ec32bfb500659cc7c02d933d9ca0a0473715ad2a399cd3c86ad76a698af9 Nov 02 23:38:07.232452 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:07.232420 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:38:07.232543 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:07.232508 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:38:07.232543 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:07.232538 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:38:08.107752 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:08.105259 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" event={"ID":"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6","Type":"ContainerStarted","Data":"cb034058ecf6a2612b791cfc08edb7a3763969aa4a7f2611abcff23b2c6abf4c"} Nov 02 23:38:08.107752 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:08.105336 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" event={"ID":"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6","Type":"ContainerStarted","Data":"d3f1ec32bfb500659cc7c02d933d9ca0a0473715ad2a399cd3c86ad76a698af9"} Nov 02 23:38:08.241958 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:08.240962 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" podStartSLOduration=2.240874902 podStartE2EDuration="2.240874902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:38:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:38:08.230598757 +0000 UTC m=+2737.199130542" watchObservedRunningTime="2026-11-02 23:38:08.240874902 +0000 UTC m=+2737.209406687" Nov 02 23:38:08.780099 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:08.780060 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" Nov 02 23:38:08.781790 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:08.781761 5076 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 02 23:38:08.782138 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:08.781816 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" podUID="18e99da1-94f3-4d3e-a6ce-ee5a43219ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 02 23:38:09.780884 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:09.780845 5076 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 02 23:38:09.782796 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:09.781457 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" podUID="18e99da1-94f3-4d3e-a6ce-ee5a43219ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 02 23:38:09.883520 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:38:09.883481 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:38:10.780429 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:10.780389 5076 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 02 23:38:10.780605 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:10.780456 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" podUID="18e99da1-94f3-4d3e-a6ce-ee5a43219ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 02 23:38:11.780974 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:11.780940 5076 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 02 23:38:11.781405 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:11.780997 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" podUID="18e99da1-94f3-4d3e-a6ce-ee5a43219ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 02 23:38:12.161124 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:12.161024 5076 generic.go:356] "Generic (PLEG): container finished" podID="3fd88117-cf15-4806-ade9-c17982846d7d" containerID="b806b7dfdb2b05430ae52c431243adad3d9c999fc29dac4f7e47874c6a5db367" exitCode=0 Nov 02 23:38:12.161124 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:12.161106 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" event={"ID":"3fd88117-cf15-4806-ade9-c17982846d7d","Type":"ContainerDied","Data":"b806b7dfdb2b05430ae52c431243adad3d9c999fc29dac4f7e47874c6a5db367"} Nov 02 23:38:12.161604 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:12.161586 5076 scope.go:122] "RemoveContainer" containerID="b806b7dfdb2b05430ae52c431243adad3d9c999fc29dac4f7e47874c6a5db367" Nov 02 23:38:12.163174 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:12.163142 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:38:12.163294 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:12.163230 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:38:12.163294 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:12.163279 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:38:12.780964 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:12.780935 5076 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 02 23:38:12.781151 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:12.780985 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" podUID="18e99da1-94f3-4d3e-a6ce-ee5a43219ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 02 23:38:13.170006 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:13.169918 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" event={"ID":"3fd88117-cf15-4806-ade9-c17982846d7d","Type":"ContainerStarted","Data":"252dc8ada1fc2473291145ed0d3ec3dd164bcd02b51b284eaebda2ba36678974"} Nov 02 23:38:13.331367 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:13.331330 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" Nov 02 23:38:13.331747 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:13.331724 5076 patch_prober.go:29] interesting pod/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 02 23:38:13.331913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:13.331873 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" podUID="3fd88117-cf15-4806-ade9-c17982846d7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 02 23:38:13.780749 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:13.780716 5076 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 02 23:38:13.780954 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:13.780774 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" podUID="18e99da1-94f3-4d3e-a6ce-ee5a43219ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 02 23:38:14.331399 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:14.331354 5076 patch_prober.go:29] interesting pod/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 02 23:38:14.332075 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:14.332038 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" podUID="3fd88117-cf15-4806-ade9-c17982846d7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 02 23:38:14.988204 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:14.988170 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" Nov 02 23:38:14.988498 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:14.988476 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" Nov 02 23:38:14.989255 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:14.989238 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" Nov 02 23:38:15.333969 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:15.333326 5076 patch_prober.go:29] interesting pod/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 02 23:38:15.333969 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:15.333434 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" podUID="3fd88117-cf15-4806-ade9-c17982846d7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 02 23:38:16.331915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:16.330990 5076 patch_prober.go:29] interesting pod/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 02 23:38:16.331915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:16.331065 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" podUID="3fd88117-cf15-4806-ade9-c17982846d7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 02 23:38:16.769678 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:16.769630 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8"] Nov 02 23:38:16.772955 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:16.772933 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" Nov 02 23:38:16.828911 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:16.828742 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-b1be95-openshift-default-dockercfg-k7jb4\"" Nov 02 23:38:17.015621 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:17.013924 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8"] Nov 02 23:38:17.095609 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:17.095509 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/974b990b-722c-427a-9328-6f78b23b63d3-workload-socket\") pod \"e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8\" (UID: \"974b990b-722c-427a-9328-6f78b23b63d3\") " pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" Nov 02 23:38:17.095609 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:17.095560 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtn24\" (UniqueName: \"kubernetes.io/projected/974b990b-722c-427a-9328-6f78b23b63d3-kube-api-access-mtn24\") pod \"e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8\" (UID: \"974b990b-722c-427a-9328-6f78b23b63d3\") " pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" Nov 02 23:38:17.095609 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:17.095595 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/974b990b-722c-427a-9328-6f78b23b63d3-istio-podinfo\") pod \"e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8\" (UID: \"974b990b-722c-427a-9328-6f78b23b63d3\") " pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" Nov 02 23:38:17.095913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:17.095670 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/974b990b-722c-427a-9328-6f78b23b63d3-istio-data\") pod \"e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8\" (UID: \"974b990b-722c-427a-9328-6f78b23b63d3\") " pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" Nov 02 23:38:17.095913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:17.095731 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/974b990b-722c-427a-9328-6f78b23b63d3-istio-envoy\") pod \"e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8\" (UID: \"974b990b-722c-427a-9328-6f78b23b63d3\") " pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" Nov 02 23:38:17.095913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:17.095755 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/974b990b-722c-427a-9328-6f78b23b63d3-istio-token\") pod \"e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8\" (UID: \"974b990b-722c-427a-9328-6f78b23b63d3\") " pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" Nov 02 23:38:17.095913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:17.095789 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/974b990b-722c-427a-9328-6f78b23b63d3-workload-certs\") pod \"e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8\" (UID: \"974b990b-722c-427a-9328-6f78b23b63d3\") " pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" Nov 02 23:38:17.095913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:17.095866 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/974b990b-722c-427a-9328-6f78b23b63d3-credential-socket\") pod \"e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8\" (UID: \"974b990b-722c-427a-9328-6f78b23b63d3\") " pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" Nov 02 23:38:17.096161 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:17.095917 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/974b990b-722c-427a-9328-6f78b23b63d3-istiod-ca-cert\") pod \"e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8\" (UID: \"974b990b-722c-427a-9328-6f78b23b63d3\") " pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" Nov 02 23:38:17.196956 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:17.196921 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/974b990b-722c-427a-9328-6f78b23b63d3-credential-socket\") pod \"e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8\" (UID: \"974b990b-722c-427a-9328-6f78b23b63d3\") " pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" Nov 02 23:38:17.196956 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:17.196967 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/974b990b-722c-427a-9328-6f78b23b63d3-istiod-ca-cert\") pod \"e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8\" (UID: \"974b990b-722c-427a-9328-6f78b23b63d3\") " pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" Nov 02 23:38:17.197223 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:17.197048 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/974b990b-722c-427a-9328-6f78b23b63d3-workload-socket\") pod \"e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8\" (UID: \"974b990b-722c-427a-9328-6f78b23b63d3\") " pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" Nov 02 23:38:17.197223 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:17.197067 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtn24\" (UniqueName: \"kubernetes.io/projected/974b990b-722c-427a-9328-6f78b23b63d3-kube-api-access-mtn24\") pod \"e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8\" (UID: \"974b990b-722c-427a-9328-6f78b23b63d3\") " pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" Nov 02 23:38:17.197223 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:17.197089 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/974b990b-722c-427a-9328-6f78b23b63d3-istio-podinfo\") pod \"e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8\" (UID: \"974b990b-722c-427a-9328-6f78b23b63d3\") " pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" Nov 02 23:38:17.197223 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:17.197140 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/974b990b-722c-427a-9328-6f78b23b63d3-istio-data\") pod \"e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8\" (UID: \"974b990b-722c-427a-9328-6f78b23b63d3\") " pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" Nov 02 23:38:17.197223 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:17.197179 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/974b990b-722c-427a-9328-6f78b23b63d3-istio-envoy\") pod \"e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8\" (UID: \"974b990b-722c-427a-9328-6f78b23b63d3\") " pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" Nov 02 23:38:17.197223 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:17.197194 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/974b990b-722c-427a-9328-6f78b23b63d3-istio-token\") pod \"e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8\" (UID: \"974b990b-722c-427a-9328-6f78b23b63d3\") " pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" Nov 02 23:38:17.197223 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:17.197214 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/974b990b-722c-427a-9328-6f78b23b63d3-workload-certs\") pod \"e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8\" (UID: \"974b990b-722c-427a-9328-6f78b23b63d3\") " pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" Nov 02 23:38:17.197621 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:17.197286 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/974b990b-722c-427a-9328-6f78b23b63d3-credential-socket\") pod \"e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8\" (UID: \"974b990b-722c-427a-9328-6f78b23b63d3\") " pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" Nov 02 23:38:17.197621 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:17.197569 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/974b990b-722c-427a-9328-6f78b23b63d3-istio-data\") pod \"e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8\" (UID: \"974b990b-722c-427a-9328-6f78b23b63d3\") " pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" Nov 02 23:38:17.197791 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:17.197770 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/974b990b-722c-427a-9328-6f78b23b63d3-istiod-ca-cert\") pod \"e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8\" (UID: \"974b990b-722c-427a-9328-6f78b23b63d3\") " pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" Nov 02 23:38:17.197863 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:17.197804 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/974b990b-722c-427a-9328-6f78b23b63d3-workload-certs\") pod \"e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8\" (UID: \"974b990b-722c-427a-9328-6f78b23b63d3\") " pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" Nov 02 23:38:17.198395 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:17.198369 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/974b990b-722c-427a-9328-6f78b23b63d3-workload-socket\") pod \"e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8\" (UID: \"974b990b-722c-427a-9328-6f78b23b63d3\") " pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" Nov 02 23:38:17.199728 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:17.199708 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/974b990b-722c-427a-9328-6f78b23b63d3-istio-envoy\") pod \"e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8\" (UID: \"974b990b-722c-427a-9328-6f78b23b63d3\") " pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" Nov 02 23:38:17.200046 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:17.200027 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/974b990b-722c-427a-9328-6f78b23b63d3-istio-podinfo\") pod \"e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8\" (UID: \"974b990b-722c-427a-9328-6f78b23b63d3\") " pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" Nov 02 23:38:17.261921 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:17.261859 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtn24\" (UniqueName: \"kubernetes.io/projected/974b990b-722c-427a-9328-6f78b23b63d3-kube-api-access-mtn24\") pod \"e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8\" (UID: \"974b990b-722c-427a-9328-6f78b23b63d3\") " pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" Nov 02 23:38:17.265945 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:17.262615 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/974b990b-722c-427a-9328-6f78b23b63d3-istio-token\") pod \"e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8\" (UID: \"974b990b-722c-427a-9328-6f78b23b63d3\") " pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" Nov 02 23:38:17.333728 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:17.333249 5076 patch_prober.go:29] interesting pod/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 02 23:38:17.333728 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:17.333326 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" podUID="3fd88117-cf15-4806-ade9-c17982846d7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 02 23:38:17.384937 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:17.384822 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" Nov 02 23:38:17.892358 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:17.892331 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8"] Nov 02 23:38:17.893507 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:38:17.893475 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod974b990b_722c_427a_9328_6f78b23b63d3.slice/crio-ad35c87ed65b2447c01124b3c7276d89d78bf562185b08c320203e6ec014f837 WatchSource:0}: Error finding container ad35c87ed65b2447c01124b3c7276d89d78bf562185b08c320203e6ec014f837: Status 404 returned error can't find the container with id ad35c87ed65b2447c01124b3c7276d89d78bf562185b08c320203e6ec014f837 Nov 02 23:38:17.896056 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:17.896025 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:38:17.896137 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:17.896090 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:38:17.896137 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:17.896120 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:38:18.202484 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:18.202449 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" event={"ID":"974b990b-722c-427a-9328-6f78b23b63d3","Type":"ContainerStarted","Data":"9517d0775eca1f38f2203a75a86b5a82bd7b5b05f6365c590a416fb3929a2e40"} Nov 02 23:38:18.202676 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:18.202513 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" event={"ID":"974b990b-722c-427a-9328-6f78b23b63d3","Type":"ContainerStarted","Data":"ad35c87ed65b2447c01124b3c7276d89d78bf562185b08c320203e6ec014f837"} Nov 02 23:38:18.332915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:18.323630 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" podStartSLOduration=2.323607823 podStartE2EDuration="2.323607823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:38:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:38:18.322267928 +0000 UTC m=+2747.290799726" watchObservedRunningTime="2026-11-02 23:38:18.323607823 +0000 UTC m=+2747.292139608" Nov 02 23:38:18.334902 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:18.333836 5076 patch_prober.go:29] interesting pod/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 02 23:38:18.335161 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:18.335129 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" podUID="3fd88117-cf15-4806-ade9-c17982846d7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 02 23:38:18.385587 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:18.385559 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" Nov 02 23:38:18.387076 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:18.387054 5076 patch_prober.go:29] interesting pod/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 02 23:38:18.387177 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:18.387103 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" podUID="974b990b-722c-427a-9328-6f78b23b63d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 02 23:38:19.335820 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:19.335783 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" Nov 02 23:38:19.336229 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:19.336100 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" Nov 02 23:38:19.336834 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:19.336815 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" Nov 02 23:38:19.389933 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:19.389907 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" Nov 02 23:38:20.215337 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:20.215305 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" Nov 02 23:38:20.216283 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:20.216261 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" Nov 02 23:38:36.743811 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:36.743773 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr"] Nov 02 23:38:36.749272 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:36.749250 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" Nov 02 23:38:36.804504 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:36.804471 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a2e3c3-353b-474d-83d4-c3ecc92d4963-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr\" (UID: \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" Nov 02 23:38:36.804654 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:36.804525 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a2e3c3-353b-474d-83d4-c3ecc92d4963-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr\" (UID: \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" Nov 02 23:38:36.804654 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:36.804613 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a2e3c3-353b-474d-83d4-c3ecc92d4963-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr\" (UID: \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" Nov 02 23:38:36.804736 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:36.804690 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a2e3c3-353b-474d-83d4-c3ecc92d4963-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr\" (UID: \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" Nov 02 23:38:36.804736 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:36.804727 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf9vn\" (UniqueName: \"kubernetes.io/projected/74a2e3c3-353b-474d-83d4-c3ecc92d4963-kube-api-access-rf9vn\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr\" (UID: \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" Nov 02 23:38:36.804812 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:36.804746 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a2e3c3-353b-474d-83d4-c3ecc92d4963-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr\" (UID: \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" Nov 02 23:38:36.804859 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:36.804837 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a2e3c3-353b-474d-83d4-c3ecc92d4963-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr\" (UID: \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" Nov 02 23:38:36.804937 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:36.804881 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a2e3c3-353b-474d-83d4-c3ecc92d4963-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr\" (UID: \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" Nov 02 23:38:36.804937 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:36.804928 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a2e3c3-353b-474d-83d4-c3ecc92d4963-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr\" (UID: \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" Nov 02 23:38:36.849205 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:36.849169 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr"] Nov 02 23:38:36.906622 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:36.906586 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a2e3c3-353b-474d-83d4-c3ecc92d4963-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr\" (UID: \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" Nov 02 23:38:36.906829 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:36.906741 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf9vn\" (UniqueName: \"kubernetes.io/projected/74a2e3c3-353b-474d-83d4-c3ecc92d4963-kube-api-access-rf9vn\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr\" (UID: \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" Nov 02 23:38:36.906829 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:36.906780 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a2e3c3-353b-474d-83d4-c3ecc92d4963-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr\" (UID: \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" Nov 02 23:38:36.907026 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:36.906828 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a2e3c3-353b-474d-83d4-c3ecc92d4963-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr\" (UID: \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" Nov 02 23:38:36.907026 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:36.906883 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a2e3c3-353b-474d-83d4-c3ecc92d4963-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr\" (UID: \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" Nov 02 23:38:36.907328 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:36.907300 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a2e3c3-353b-474d-83d4-c3ecc92d4963-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr\" (UID: \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" Nov 02 23:38:36.907454 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:36.907348 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a2e3c3-353b-474d-83d4-c3ecc92d4963-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr\" (UID: \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" Nov 02 23:38:36.907454 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:36.907407 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a2e3c3-353b-474d-83d4-c3ecc92d4963-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr\" (UID: \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" Nov 02 23:38:36.907571 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:36.907523 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a2e3c3-353b-474d-83d4-c3ecc92d4963-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr\" (UID: \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" Nov 02 23:38:36.907629 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:36.907597 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a2e3c3-353b-474d-83d4-c3ecc92d4963-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr\" (UID: \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" Nov 02 23:38:36.907690 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:36.907651 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a2e3c3-353b-474d-83d4-c3ecc92d4963-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr\" (UID: \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" Nov 02 23:38:36.908131 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:36.908102 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a2e3c3-353b-474d-83d4-c3ecc92d4963-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr\" (UID: \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" Nov 02 23:38:36.908430 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:36.908403 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a2e3c3-353b-474d-83d4-c3ecc92d4963-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr\" (UID: \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" Nov 02 23:38:36.908674 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:36.908649 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a2e3c3-353b-474d-83d4-c3ecc92d4963-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr\" (UID: \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" Nov 02 23:38:36.909938 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:36.909915 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a2e3c3-353b-474d-83d4-c3ecc92d4963-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr\" (UID: \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" Nov 02 23:38:36.910038 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:36.909950 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a2e3c3-353b-474d-83d4-c3ecc92d4963-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr\" (UID: \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" Nov 02 23:38:37.033668 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:37.030648 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf9vn\" (UniqueName: \"kubernetes.io/projected/74a2e3c3-353b-474d-83d4-c3ecc92d4963-kube-api-access-rf9vn\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr\" (UID: \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" Nov 02 23:38:37.048941 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:37.046236 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a2e3c3-353b-474d-83d4-c3ecc92d4963-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr\" (UID: \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" Nov 02 23:38:37.064289 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:37.063885 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" Nov 02 23:38:37.508372 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:37.508337 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr"] Nov 02 23:38:37.509661 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:38:37.509629 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a2e3c3_353b_474d_83d4_c3ecc92d4963.slice/crio-f8bb56b2af7fe491ca3ddec67f107176ffe907a67a9aa39b3dbda7ca8c7cdea5 WatchSource:0}: Error finding container f8bb56b2af7fe491ca3ddec67f107176ffe907a67a9aa39b3dbda7ca8c7cdea5: Status 404 returned error can't find the container with id f8bb56b2af7fe491ca3ddec67f107176ffe907a67a9aa39b3dbda7ca8c7cdea5 Nov 02 23:38:37.512954 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:37.512921 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:38:37.513039 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:37.512990 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:38:37.513039 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:37.513020 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:38:38.405452 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:38.405362 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" event={"ID":"74a2e3c3-353b-474d-83d4-c3ecc92d4963","Type":"ContainerStarted","Data":"b4b7c00de6bd9fd7e010a98629a176f99ded52e69f3061fb7b659154b94f8900"} Nov 02 23:38:38.406044 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:38.406017 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" event={"ID":"74a2e3c3-353b-474d-83d4-c3ecc92d4963","Type":"ContainerStarted","Data":"f8bb56b2af7fe491ca3ddec67f107176ffe907a67a9aa39b3dbda7ca8c7cdea5"} Nov 02 23:38:38.566268 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:38.566210 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" podStartSLOduration=2.566187309 podStartE2EDuration="2.566187309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:38:38.564257336 +0000 UTC m=+2767.532789169" watchObservedRunningTime="2026-11-02 23:38:38.566187309 +0000 UTC m=+2767.534719093" Nov 02 23:38:39.064425 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:39.064388 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" Nov 02 23:38:39.065695 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:39.065670 5076 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 02 23:38:39.065817 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:39.065717 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" podUID="74a2e3c3-353b-474d-83d4-c3ecc92d4963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 02 23:38:39.891989 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:38:39.891952 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:38:40.078918 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:40.077708 5076 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 02 23:38:40.078918 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:40.077772 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" podUID="74a2e3c3-353b-474d-83d4-c3ecc92d4963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 02 23:38:41.065915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:41.065078 5076 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 02 23:38:41.065915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:41.065141 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" podUID="74a2e3c3-353b-474d-83d4-c3ecc92d4963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 02 23:38:42.065375 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:42.065340 5076 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 02 23:38:42.065560 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:42.065409 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" podUID="74a2e3c3-353b-474d-83d4-c3ecc92d4963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 02 23:38:43.065374 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:43.065332 5076 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 02 23:38:43.065994 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:43.065404 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" podUID="74a2e3c3-353b-474d-83d4-c3ecc92d4963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 02 23:38:44.065141 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:44.065100 5076 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 02 23:38:44.065319 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:44.065168 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" podUID="74a2e3c3-353b-474d-83d4-c3ecc92d4963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 02 23:38:45.070342 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:45.070311 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" Nov 02 23:38:45.070798 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:45.070776 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" Nov 02 23:38:45.071582 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:45.071560 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" Nov 02 23:38:47.515314 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:47.515281 5076 generic.go:356] "Generic (PLEG): container finished" podID="51ba5367-449d-4292-9868-6ff46c0f94d3" containerID="7c99d708208871f9cca3b3e1f9608110d458c6fd09204a5433513332f9d7238f" exitCode=0 Nov 02 23:38:47.515812 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:47.515332 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" event={"ID":"51ba5367-449d-4292-9868-6ff46c0f94d3","Type":"ContainerDied","Data":"7c99d708208871f9cca3b3e1f9608110d458c6fd09204a5433513332f9d7238f"} Nov 02 23:38:47.515812 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:47.515754 5076 scope.go:122] "RemoveContainer" containerID="7c99d708208871f9cca3b3e1f9608110d458c6fd09204a5433513332f9d7238f" Nov 02 23:38:47.517320 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:47.517288 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:38:47.517406 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:47.517358 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:38:47.517406 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:47.517394 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:38:47.771520 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:47.771438 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt"] Nov 02 23:38:47.775548 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:47.775520 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt" Nov 02 23:38:47.826157 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:47.826126 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-b4d53e-serving-cert\"" Nov 02 23:38:47.826519 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:47.826489 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt"] Nov 02 23:38:47.936585 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:47.936543 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79ab1c7e-f257-492f-ac2f-dfa72b75d711-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt\" (UID: \"79ab1c7e-f257-492f-ac2f-dfa72b75d711\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt" Nov 02 23:38:47.936762 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:47.936688 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8ctd\" (UniqueName: \"kubernetes.io/projected/79ab1c7e-f257-492f-ac2f-dfa72b75d711-kube-api-access-k8ctd\") pod \"maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt\" (UID: \"79ab1c7e-f257-492f-ac2f-dfa72b75d711\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt" Nov 02 23:38:47.936869 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:47.936847 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79ab1c7e-f257-492f-ac2f-dfa72b75d711-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt\" (UID: \"79ab1c7e-f257-492f-ac2f-dfa72b75d711\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt" Nov 02 23:38:47.937037 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:47.937012 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ab1c7e-f257-492f-ac2f-dfa72b75d711-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt\" (UID: \"79ab1c7e-f257-492f-ac2f-dfa72b75d711\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt" Nov 02 23:38:48.038792 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:48.038678 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79ab1c7e-f257-492f-ac2f-dfa72b75d711-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt\" (UID: \"79ab1c7e-f257-492f-ac2f-dfa72b75d711\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt" Nov 02 23:38:48.039156 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:48.039129 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8ctd\" (UniqueName: \"kubernetes.io/projected/79ab1c7e-f257-492f-ac2f-dfa72b75d711-kube-api-access-k8ctd\") pod \"maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt\" (UID: \"79ab1c7e-f257-492f-ac2f-dfa72b75d711\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt" Nov 02 23:38:48.039340 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:48.039322 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79ab1c7e-f257-492f-ac2f-dfa72b75d711-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt\" (UID: \"79ab1c7e-f257-492f-ac2f-dfa72b75d711\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt" Nov 02 23:38:48.048201 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:48.047031 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ab1c7e-f257-492f-ac2f-dfa72b75d711-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt\" (UID: \"79ab1c7e-f257-492f-ac2f-dfa72b75d711\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt" Nov 02 23:38:48.048201 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:48.048136 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ab1c7e-f257-492f-ac2f-dfa72b75d711-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt\" (UID: \"79ab1c7e-f257-492f-ac2f-dfa72b75d711\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt" Nov 02 23:38:48.051151 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:48.051124 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79ab1c7e-f257-492f-ac2f-dfa72b75d711-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt\" (UID: \"79ab1c7e-f257-492f-ac2f-dfa72b75d711\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt" Nov 02 23:38:48.051283 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:48.051124 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79ab1c7e-f257-492f-ac2f-dfa72b75d711-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt\" (UID: \"79ab1c7e-f257-492f-ac2f-dfa72b75d711\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt" Nov 02 23:38:48.102427 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:48.102390 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8ctd\" (UniqueName: \"kubernetes.io/projected/79ab1c7e-f257-492f-ac2f-dfa72b75d711-kube-api-access-k8ctd\") pod \"maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt\" (UID: \"79ab1c7e-f257-492f-ac2f-dfa72b75d711\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt" Nov 02 23:38:48.104093 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:48.104067 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt" Nov 02 23:38:48.540134 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:48.540101 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" event={"ID":"51ba5367-449d-4292-9868-6ff46c0f94d3","Type":"ContainerStarted","Data":"e813b08762b3f2a4ca1fa2678ee9954751106248cc3d6b80dc12e38a4d26d578"} Nov 02 23:38:48.608671 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:48.608644 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt"] Nov 02 23:38:49.365847 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:49.365811 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" Nov 02 23:38:49.366032 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:49.365975 5076 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 02 23:38:49.366032 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:49.366024 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" podUID="51ba5367-449d-4292-9868-6ff46c0f94d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 02 23:38:49.547741 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:49.547705 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt" event={"ID":"79ab1c7e-f257-492f-ac2f-dfa72b75d711","Type":"ContainerStarted","Data":"2e04c4fb316228324ed5f44de38d6f95f0e4650a22c5fae1adc4279ea47362f1"} Nov 02 23:38:50.365867 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:50.365829 5076 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 02 23:38:50.366056 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:50.365906 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" podUID="51ba5367-449d-4292-9868-6ff46c0f94d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 02 23:38:50.555052 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:50.555016 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt" event={"ID":"79ab1c7e-f257-492f-ac2f-dfa72b75d711","Type":"ContainerStarted","Data":"0dd4d4e9542132b2cd759f9b529cb21497c00c9436b9e8e223d384fd4c524af1"} Nov 02 23:38:50.555458 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:50.555134 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt" Nov 02 23:38:50.685706 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:50.685648 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt" podStartSLOduration=2.8139195900000002 podStartE2EDuration="3.685634622s" totalImagesPullingTime="871.715032ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:38:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:38:50.683818117 +0000 UTC m=+2779.652349903" watchObservedRunningTime="2026-11-02 23:38:50.685634622 +0000 UTC m=+2779.654166406" Nov 02 23:38:51.366417 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:51.366379 5076 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 02 23:38:51.366595 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:51.366452 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" podUID="51ba5367-449d-4292-9868-6ff46c0f94d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 02 23:38:52.366429 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:52.366395 5076 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 02 23:38:52.366794 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:52.366454 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" podUID="51ba5367-449d-4292-9868-6ff46c0f94d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 02 23:38:53.366725 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:53.366689 5076 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 02 23:38:53.367149 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:53.366741 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" podUID="51ba5367-449d-4292-9868-6ff46c0f94d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 02 23:38:54.370617 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:54.370583 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" Nov 02 23:38:54.371067 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:54.370985 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" Nov 02 23:38:54.371753 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:54.371732 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" Nov 02 23:38:56.566160 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:56.566130 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt" Nov 02 23:38:57.785790 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:57.785760 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq"] Nov 02 23:38:57.791190 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:57.791170 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq" Nov 02 23:38:57.845819 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:57.845785 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq"] Nov 02 23:38:57.888022 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:57.887982 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-28f82c-serving-cert\"" Nov 02 23:38:57.908629 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:57.908592 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62d85249-11d4-47a6-955d-978815d12fd9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq\" (UID: \"62d85249-11d4-47a6-955d-978815d12fd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq" Nov 02 23:38:57.908773 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:57.908636 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62d85249-11d4-47a6-955d-978815d12fd9-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq\" (UID: \"62d85249-11d4-47a6-955d-978815d12fd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq" Nov 02 23:38:57.908773 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:57.908662 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62d85249-11d4-47a6-955d-978815d12fd9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq\" (UID: \"62d85249-11d4-47a6-955d-978815d12fd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq" Nov 02 23:38:57.908850 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:57.908828 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbvb9\" (UniqueName: \"kubernetes.io/projected/62d85249-11d4-47a6-955d-978815d12fd9-kube-api-access-rbvb9\") pod \"maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq\" (UID: \"62d85249-11d4-47a6-955d-978815d12fd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq" Nov 02 23:38:58.010401 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:58.010362 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62d85249-11d4-47a6-955d-978815d12fd9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq\" (UID: \"62d85249-11d4-47a6-955d-978815d12fd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq" Nov 02 23:38:58.010617 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:58.010420 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62d85249-11d4-47a6-955d-978815d12fd9-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq\" (UID: \"62d85249-11d4-47a6-955d-978815d12fd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq" Nov 02 23:38:58.010617 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:58.010458 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62d85249-11d4-47a6-955d-978815d12fd9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq\" (UID: \"62d85249-11d4-47a6-955d-978815d12fd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq" Nov 02 23:38:58.010617 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:58.010582 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbvb9\" (UniqueName: \"kubernetes.io/projected/62d85249-11d4-47a6-955d-978815d12fd9-kube-api-access-rbvb9\") pod \"maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq\" (UID: \"62d85249-11d4-47a6-955d-978815d12fd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq" Nov 02 23:38:58.010777 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:38:58.010666 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-28f82c-serving-cert: secret "maas-api-e2e-shared-a-w3-28f82c-serving-cert" not found Nov 02 23:38:58.010777 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:38:58.010745 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62d85249-11d4-47a6-955d-978815d12fd9-maas-api-tls podName:62d85249-11d4-47a6-955d-978815d12fd9 nodeName:}" failed. No retries permitted until 2026-11-02 23:38:58.510725901 +0000 UTC m=+2787.479257663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/62d85249-11d4-47a6-955d-978815d12fd9-maas-api-tls") pod "maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq" (UID: "62d85249-11d4-47a6-955d-978815d12fd9") : secret "maas-api-e2e-shared-a-w3-28f82c-serving-cert" not found Nov 02 23:38:58.011176 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:58.011152 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62d85249-11d4-47a6-955d-978815d12fd9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq\" (UID: \"62d85249-11d4-47a6-955d-978815d12fd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq" Nov 02 23:38:58.013006 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:58.012985 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62d85249-11d4-47a6-955d-978815d12fd9-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq\" (UID: \"62d85249-11d4-47a6-955d-978815d12fd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq" Nov 02 23:38:58.065337 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:58.065255 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbvb9\" (UniqueName: \"kubernetes.io/projected/62d85249-11d4-47a6-955d-978815d12fd9-kube-api-access-rbvb9\") pod \"maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq\" (UID: \"62d85249-11d4-47a6-955d-978815d12fd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq" Nov 02 23:38:58.524233 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:58.524192 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62d85249-11d4-47a6-955d-978815d12fd9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq\" (UID: \"62d85249-11d4-47a6-955d-978815d12fd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq" Nov 02 23:38:58.526672 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:58.526651 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62d85249-11d4-47a6-955d-978815d12fd9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq\" (UID: \"62d85249-11d4-47a6-955d-978815d12fd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq" Nov 02 23:38:58.705012 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:58.704961 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq" Nov 02 23:38:59.090364 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:59.090330 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq"] Nov 02 23:38:59.090827 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:38:59.090803 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62d85249_11d4_47a6_955d_978815d12fd9.slice/crio-3902aab50cf16528aa9182160675cb538d03bdbd23d328f47a416a2a7144fc6a WatchSource:0}: Error finding container 3902aab50cf16528aa9182160675cb538d03bdbd23d328f47a416a2a7144fc6a: Status 404 returned error can't find the container with id 3902aab50cf16528aa9182160675cb538d03bdbd23d328f47a416a2a7144fc6a Nov 02 23:38:59.611972 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:38:59.611927 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq" event={"ID":"62d85249-11d4-47a6-955d-978815d12fd9","Type":"ContainerStarted","Data":"3902aab50cf16528aa9182160675cb538d03bdbd23d328f47a416a2a7144fc6a"} Nov 02 23:39:00.619864 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:00.619826 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq" event={"ID":"62d85249-11d4-47a6-955d-978815d12fd9","Type":"ContainerStarted","Data":"6de065b514eeccb4211af4e0fb0421669f61a0f8c28d1c076aa990ecd668029c"} Nov 02 23:39:00.620251 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:00.619938 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq" Nov 02 23:39:00.731759 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:00.731706 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq" podStartSLOduration=2.796508323 podStartE2EDuration="3.73169243s" totalImagesPullingTime="935.184107ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:38:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:39:00.728494839 +0000 UTC m=+2789.697026622" watchObservedRunningTime="2026-11-02 23:39:00.73169243 +0000 UTC m=+2789.700224234" Nov 02 23:39:06.638139 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:06.638106 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq" Nov 02 23:39:09.883863 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:39:09.883821 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:39:12.012519 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:12.011187 5076 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": read tcp 10.132.0.2:36982->10.132.0.159:15021: read: connection reset by peer" start-of-body= Nov 02 23:39:12.012519 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:12.011264 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" podUID="4deff109-12f0-42e8-8908-508e7f6eec68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": read tcp 10.132.0.2:36982->10.132.0.159:15021: read: connection reset by peer" Nov 02 23:39:12.699123 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:12.699089 5076 generic.go:356] "Generic (PLEG): container finished" podID="4deff109-12f0-42e8-8908-508e7f6eec68" containerID="e64290c914e0b51b805d0321fa2662f5f72e4e7b44f8e746614271b05e4e5b50" exitCode=0 Nov 02 23:39:12.699298 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:12.699128 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" event={"ID":"4deff109-12f0-42e8-8908-508e7f6eec68","Type":"ContainerDied","Data":"e64290c914e0b51b805d0321fa2662f5f72e4e7b44f8e746614271b05e4e5b50"} Nov 02 23:39:12.699545 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:12.699531 5076 scope.go:122] "RemoveContainer" containerID="e64290c914e0b51b805d0321fa2662f5f72e4e7b44f8e746614271b05e4e5b50" Nov 02 23:39:12.701049 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:12.701023 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:39:12.701111 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:12.701087 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:39:12.701147 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:12.701120 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:39:13.707429 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:13.707396 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" event={"ID":"4deff109-12f0-42e8-8908-508e7f6eec68","Type":"ContainerStarted","Data":"da3b04c80a5932782a81cab47be5b6c880040878182c7a1ba0f9b538dc564ca3"} Nov 02 23:39:13.834998 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:13.834294 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" Nov 02 23:39:13.834998 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:13.834417 5076 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 02 23:39:13.834998 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:13.834458 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" podUID="4deff109-12f0-42e8-8908-508e7f6eec68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 02 23:39:14.834241 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:14.834207 5076 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 02 23:39:14.834756 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:14.834269 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" podUID="4deff109-12f0-42e8-8908-508e7f6eec68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 02 23:39:15.834220 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:15.834185 5076 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 02 23:39:15.834430 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:15.834245 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" podUID="4deff109-12f0-42e8-8908-508e7f6eec68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 02 23:39:16.834217 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:16.834175 5076 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 02 23:39:16.834448 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:16.834238 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" podUID="4deff109-12f0-42e8-8908-508e7f6eec68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 02 23:39:17.834250 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:17.834200 5076 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 02 23:39:17.834454 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:17.834265 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" podUID="4deff109-12f0-42e8-8908-508e7f6eec68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 02 23:39:18.832224 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:18.832186 5076 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 02 23:39:18.832441 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:18.832248 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" podUID="4deff109-12f0-42e8-8908-508e7f6eec68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 02 23:39:19.837313 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:19.837279 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" Nov 02 23:39:19.837800 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:19.837577 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" Nov 02 23:39:19.838445 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:19.838412 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" Nov 02 23:39:39.955171 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:39:39.955112 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:39:40.049724 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:40.049686 5076 generic.go:356] "Generic (PLEG): container finished" podID="974b990b-722c-427a-9328-6f78b23b63d3" containerID="9517d0775eca1f38f2203a75a86b5a82bd7b5b05f6365c590a416fb3929a2e40" exitCode=0 Nov 02 23:39:40.049982 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:40.049779 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" event={"ID":"974b990b-722c-427a-9328-6f78b23b63d3","Type":"ContainerDied","Data":"9517d0775eca1f38f2203a75a86b5a82bd7b5b05f6365c590a416fb3929a2e40"} Nov 02 23:39:40.050353 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:40.050330 5076 scope.go:122] "RemoveContainer" containerID="9517d0775eca1f38f2203a75a86b5a82bd7b5b05f6365c590a416fb3929a2e40" Nov 02 23:39:40.053359 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:40.052984 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:39:40.053359 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:40.053091 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:39:40.053359 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:40.053141 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:39:41.057696 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:41.057656 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" event={"ID":"974b990b-722c-427a-9328-6f78b23b63d3","Type":"ContainerStarted","Data":"e35c61b916989d98518ac882c21fe90487e47ac28b122ac22194eb760aece8fd"} Nov 02 23:39:41.385995 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:41.385846 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" Nov 02 23:39:41.386158 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:41.386069 5076 patch_prober.go:29] interesting pod/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 02 23:39:41.386158 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:41.386136 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" podUID="974b990b-722c-427a-9328-6f78b23b63d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 02 23:39:42.386086 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:42.386052 5076 patch_prober.go:29] interesting pod/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 02 23:39:42.386469 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:42.386104 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" podUID="974b990b-722c-427a-9328-6f78b23b63d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 02 23:39:43.385398 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:43.385359 5076 patch_prober.go:29] interesting pod/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 02 23:39:43.385571 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:43.385438 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" podUID="974b990b-722c-427a-9328-6f78b23b63d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 02 23:39:44.386136 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:44.386096 5076 patch_prober.go:29] interesting pod/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 02 23:39:44.386650 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:44.386165 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" podUID="974b990b-722c-427a-9328-6f78b23b63d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 02 23:39:45.385953 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:45.385920 5076 patch_prober.go:29] interesting pod/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 02 23:39:45.386140 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:45.385972 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" podUID="974b990b-722c-427a-9328-6f78b23b63d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 02 23:39:46.390002 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:46.389973 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" Nov 02 23:39:46.390428 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:46.390364 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" Nov 02 23:39:46.391092 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:46.391071 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" Nov 02 23:39:58.844542 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:58.844496 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j"] Nov 02 23:39:58.848128 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:58.848110 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" Nov 02 23:39:58.908879 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:58.908844 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-a51378-openshift-default-dockercfg-qfzht\"" Nov 02 23:39:58.953922 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:58.945483 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-workload-socket\") pod \"e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j\" (UID: \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\") " pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" Nov 02 23:39:58.954410 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:58.954378 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-istio-data\") pod \"e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j\" (UID: \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\") " pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" Nov 02 23:39:58.954631 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:58.954612 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp7jt\" (UniqueName: \"kubernetes.io/projected/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-kube-api-access-zp7jt\") pod \"e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j\" (UID: \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\") " pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" Nov 02 23:39:58.954777 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:58.954762 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-istiod-ca-cert\") pod \"e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j\" (UID: \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\") " pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" Nov 02 23:39:58.954950 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:58.954933 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-credential-socket\") pod \"e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j\" (UID: \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\") " pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" Nov 02 23:39:58.960922 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:58.955171 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-istio-token\") pod \"e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j\" (UID: \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\") " pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" Nov 02 23:39:59.002915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:58.986996 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-istio-podinfo\") pod \"e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j\" (UID: \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\") " pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" Nov 02 23:39:59.002915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:58.987083 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-workload-certs\") pod \"e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j\" (UID: \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\") " pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" Nov 02 23:39:59.002915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:58.987137 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-istio-envoy\") pod \"e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j\" (UID: \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\") " pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" Nov 02 23:39:59.090704 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:59.090660 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp7jt\" (UniqueName: \"kubernetes.io/projected/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-kube-api-access-zp7jt\") pod \"e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j\" (UID: \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\") " pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" Nov 02 23:39:59.093094 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:59.092353 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-istiod-ca-cert\") pod \"e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j\" (UID: \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\") " pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" Nov 02 23:39:59.094476 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:59.094381 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-credential-socket\") pod \"e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j\" (UID: \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\") " pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" Nov 02 23:39:59.096846 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:59.096787 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-istio-token\") pod \"e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j\" (UID: \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\") " pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" Nov 02 23:39:59.097403 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:59.097380 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-istio-podinfo\") pod \"e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j\" (UID: \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\") " pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" Nov 02 23:39:59.098068 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:59.098047 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-workload-certs\") pod \"e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j\" (UID: \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\") " pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" Nov 02 23:39:59.098539 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:59.098523 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-istio-envoy\") pod \"e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j\" (UID: \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\") " pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" Nov 02 23:39:59.099099 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:59.099077 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-workload-socket\") pod \"e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j\" (UID: \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\") " pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" Nov 02 23:39:59.099606 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:59.099588 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-istio-data\") pod \"e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j\" (UID: \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\") " pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" Nov 02 23:39:59.100239 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:59.094945 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-credential-socket\") pod \"e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j\" (UID: \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\") " pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" Nov 02 23:39:59.100634 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:59.092112 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j"] Nov 02 23:39:59.100831 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:59.094095 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-istiod-ca-cert\") pod \"e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j\" (UID: \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\") " pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" Nov 02 23:39:59.100965 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:59.099473 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-workload-socket\") pod \"e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j\" (UID: \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\") " pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" Nov 02 23:39:59.101066 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:59.098451 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-workload-certs\") pod \"e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j\" (UID: \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\") " pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" Nov 02 23:39:59.101176 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:59.100005 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-istio-data\") pod \"e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j\" (UID: \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\") " pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" Nov 02 23:39:59.171925 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:59.166698 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp7jt\" (UniqueName: \"kubernetes.io/projected/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-kube-api-access-zp7jt\") pod \"e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j\" (UID: \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\") " pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" Nov 02 23:39:59.185773 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:59.185730 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-istio-podinfo\") pod \"e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j\" (UID: \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\") " pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" Nov 02 23:39:59.188985 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:59.188939 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-istio-token\") pod \"e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j\" (UID: \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\") " pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" Nov 02 23:39:59.196245 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:59.196219 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-istio-envoy\") pod \"e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j\" (UID: \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\") " pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" Nov 02 23:39:59.242622 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:59.242585 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" Nov 02 23:39:59.703657 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:59.702821 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j"] Nov 02 23:39:59.703657 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:59.703101 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:39:59.703657 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:59.703184 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:39:59.703657 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:39:59.703224 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:40:00.272362 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:00.272321 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" event={"ID":"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e","Type":"ContainerStarted","Data":"ebc686f9abf6989bad70a068d8bc3eade1b424a6ee79d59586fb29d8850cb6df"} Nov 02 23:40:00.272748 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:00.272400 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" event={"ID":"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e","Type":"ContainerStarted","Data":"1b2da61166ae4790ff50115a5c1b0fa9a406a0966093db1b967503b33f59824e"} Nov 02 23:40:00.384823 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:00.384764 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" podStartSLOduration=2.38475074 podStartE2EDuration="2.38475074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:39:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:40:00.382290341 +0000 UTC m=+2849.350822137" watchObservedRunningTime="2026-11-02 23:40:00.38475074 +0000 UTC m=+2849.353282524" Nov 02 23:40:01.243096 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:01.243059 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" Nov 02 23:40:01.249033 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:01.249008 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" Nov 02 23:40:01.278275 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:01.278244 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" Nov 02 23:40:01.279331 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:01.279311 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" Nov 02 23:40:04.369490 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:04.369448 5076 generic.go:356] "Generic (PLEG): container finished" podID="ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2" containerID="574dbc24446700e0bb5156ce4e1418b3041c05c7110bd1214dcaf87b8e76a31e" exitCode=0 Nov 02 23:40:04.370220 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:04.370192 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" event={"ID":"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2","Type":"ContainerDied","Data":"574dbc24446700e0bb5156ce4e1418b3041c05c7110bd1214dcaf87b8e76a31e"} Nov 02 23:40:04.395914 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:04.370817 5076 scope.go:122] "RemoveContainer" containerID="574dbc24446700e0bb5156ce4e1418b3041c05c7110bd1214dcaf87b8e76a31e" Nov 02 23:40:04.399153 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:04.399113 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:40:04.399803 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:04.399776 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:40:04.400038 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:04.400017 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:40:05.378171 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:05.378139 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" event={"ID":"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2","Type":"ContainerStarted","Data":"a5dc776994907dbfa0df23116eba8ee961284b56dba108f127bfc4dbf0664655"} Nov 02 23:40:05.446129 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:05.446098 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" Nov 02 23:40:05.446694 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:05.446672 5076 patch_prober.go:29] interesting pod/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 02 23:40:05.446753 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:05.446720 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" podUID="ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 02 23:40:06.445614 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:06.445572 5076 patch_prober.go:29] interesting pod/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 02 23:40:06.446162 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:06.445652 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" podUID="ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 02 23:40:07.446383 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:07.446343 5076 patch_prober.go:29] interesting pod/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 02 23:40:07.446787 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:07.446401 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" podUID="ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 02 23:40:08.446453 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:08.446413 5076 patch_prober.go:29] interesting pod/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 02 23:40:08.446997 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:08.446496 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" podUID="ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 02 23:40:09.446260 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:09.446223 5076 patch_prober.go:29] interesting pod/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 02 23:40:09.446472 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:09.446291 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" podUID="ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 02 23:40:09.885841 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:40:09.885761 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:40:10.445749 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:10.445713 5076 patch_prober.go:29] interesting pod/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 02 23:40:10.445966 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:10.445776 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" podUID="ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 02 23:40:11.451126 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:11.451097 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" Nov 02 23:40:11.451540 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:11.451517 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" Nov 02 23:40:11.452249 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:11.452230 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" Nov 02 23:40:39.885422 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:40:39.885376 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:40:41.944975 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:41.944940 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s"] Nov 02 23:40:41.950385 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:41.950363 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s" Nov 02 23:40:42.047840 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:42.047803 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-fbec82-serving-cert\"" Nov 02 23:40:42.051366 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:42.051336 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s"] Nov 02 23:40:42.073988 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:42.073958 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/272994cd-ddb4-40ae-8cdd-5a91656a7603-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s\" (UID: \"272994cd-ddb4-40ae-8cdd-5a91656a7603\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s" Nov 02 23:40:42.074128 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:42.074000 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn4c9\" (UniqueName: \"kubernetes.io/projected/272994cd-ddb4-40ae-8cdd-5a91656a7603-kube-api-access-cn4c9\") pod \"maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s\" (UID: \"272994cd-ddb4-40ae-8cdd-5a91656a7603\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s" Nov 02 23:40:42.074179 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:42.074157 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/272994cd-ddb4-40ae-8cdd-5a91656a7603-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s\" (UID: \"272994cd-ddb4-40ae-8cdd-5a91656a7603\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s" Nov 02 23:40:42.074257 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:42.074243 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/272994cd-ddb4-40ae-8cdd-5a91656a7603-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s\" (UID: \"272994cd-ddb4-40ae-8cdd-5a91656a7603\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s" Nov 02 23:40:42.175811 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:42.175776 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/272994cd-ddb4-40ae-8cdd-5a91656a7603-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s\" (UID: \"272994cd-ddb4-40ae-8cdd-5a91656a7603\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s" Nov 02 23:40:42.176013 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:42.175820 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn4c9\" (UniqueName: \"kubernetes.io/projected/272994cd-ddb4-40ae-8cdd-5a91656a7603-kube-api-access-cn4c9\") pod \"maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s\" (UID: \"272994cd-ddb4-40ae-8cdd-5a91656a7603\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s" Nov 02 23:40:42.176013 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:40:42.175957 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbec82-serving-cert: secret "maas-api-e2e-shared-b-w2-fbec82-serving-cert" not found Nov 02 23:40:42.176013 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:42.175968 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/272994cd-ddb4-40ae-8cdd-5a91656a7603-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s\" (UID: \"272994cd-ddb4-40ae-8cdd-5a91656a7603\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s" Nov 02 23:40:42.176140 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:40:42.176034 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/272994cd-ddb4-40ae-8cdd-5a91656a7603-maas-api-tls podName:272994cd-ddb4-40ae-8cdd-5a91656a7603 nodeName:}" failed. No retries permitted until 2026-11-02 23:40:42.676012645 +0000 UTC m=+2891.644544427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/272994cd-ddb4-40ae-8cdd-5a91656a7603-maas-api-tls") pod "maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s" (UID: "272994cd-ddb4-40ae-8cdd-5a91656a7603") : secret "maas-api-e2e-shared-b-w2-fbec82-serving-cert" not found Nov 02 23:40:42.176140 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:42.176099 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/272994cd-ddb4-40ae-8cdd-5a91656a7603-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s\" (UID: \"272994cd-ddb4-40ae-8cdd-5a91656a7603\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s" Nov 02 23:40:42.176707 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:42.176684 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/272994cd-ddb4-40ae-8cdd-5a91656a7603-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s\" (UID: \"272994cd-ddb4-40ae-8cdd-5a91656a7603\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s" Nov 02 23:40:42.178563 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:42.178542 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/272994cd-ddb4-40ae-8cdd-5a91656a7603-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s\" (UID: \"272994cd-ddb4-40ae-8cdd-5a91656a7603\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s" Nov 02 23:40:42.264987 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:42.264872 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn4c9\" (UniqueName: \"kubernetes.io/projected/272994cd-ddb4-40ae-8cdd-5a91656a7603-kube-api-access-cn4c9\") pod \"maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s\" (UID: \"272994cd-ddb4-40ae-8cdd-5a91656a7603\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s" Nov 02 23:40:42.719013 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:42.718975 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/272994cd-ddb4-40ae-8cdd-5a91656a7603-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s\" (UID: \"272994cd-ddb4-40ae-8cdd-5a91656a7603\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s" Nov 02 23:40:42.721626 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:42.721592 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/272994cd-ddb4-40ae-8cdd-5a91656a7603-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s\" (UID: \"272994cd-ddb4-40ae-8cdd-5a91656a7603\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s" Nov 02 23:40:42.864645 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:42.864612 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s" Nov 02 23:40:43.236224 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:40:43.236190 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod272994cd_ddb4_40ae_8cdd_5a91656a7603.slice/crio-e5fe0a67474e9c4866e6940962d96ad18d53a9cae15bfaf77e506c3c6970d483 WatchSource:0}: Error finding container e5fe0a67474e9c4866e6940962d96ad18d53a9cae15bfaf77e506c3c6970d483: Status 404 returned error can't find the container with id e5fe0a67474e9c4866e6940962d96ad18d53a9cae15bfaf77e506c3c6970d483 Nov 02 23:40:43.237382 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:43.237356 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s"] Nov 02 23:40:43.844377 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:43.844343 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s" event={"ID":"272994cd-ddb4-40ae-8cdd-5a91656a7603","Type":"ContainerStarted","Data":"e5fe0a67474e9c4866e6940962d96ad18d53a9cae15bfaf77e506c3c6970d483"} Nov 02 23:40:44.852191 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:44.852150 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s" event={"ID":"272994cd-ddb4-40ae-8cdd-5a91656a7603","Type":"ContainerStarted","Data":"408093d5816067a69aa16ffc85e6fa0d12e3764989350ea7bc7b24d67e54134d"} Nov 02 23:40:44.852606 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:44.852293 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s" Nov 02 23:40:44.962066 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:44.962012 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s" podStartSLOduration=2.997017632 podStartE2EDuration="3.9619982s" totalImagesPullingTime="964.980568ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:40:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:40:44.960665278 +0000 UTC m=+2893.929197062" watchObservedRunningTime="2026-11-02 23:40:44.9619982 +0000 UTC m=+2893.930530019" Nov 02 23:40:50.865292 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:40:50.865260 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s" Nov 02 23:41:09.884676 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:41:09.884640 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:41:22.152413 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:22.152336 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4"] Nov 02 23:41:22.152943 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:22.152511 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4" podUID="5e316f95-0005-44bd-a11b-8b04b3e32297" containerName="maas-api" containerID="cri-o://ca42ef19bab30ca0be9870bf41c21b8ace331ebeb55ecfca5db6c87a26b37dc6" gracePeriod=30 Nov 02 23:41:22.478678 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:22.478651 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4" Nov 02 23:41:22.622505 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:22.622464 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e316f95-0005-44bd-a11b-8b04b3e32297-kube-api-access-hq8dv\" (UniqueName: \"kubernetes.io/projected/5e316f95-0005-44bd-a11b-8b04b3e32297-kube-api-access-hq8dv\") pod \"5e316f95-0005-44bd-a11b-8b04b3e32297\" (UID: \"5e316f95-0005-44bd-a11b-8b04b3e32297\") " Nov 02 23:41:22.622769 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:22.622751 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e316f95-0005-44bd-a11b-8b04b3e32297-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e316f95-0005-44bd-a11b-8b04b3e32297-maas-api-tls\") pod \"5e316f95-0005-44bd-a11b-8b04b3e32297\" (UID: \"5e316f95-0005-44bd-a11b-8b04b3e32297\") " Nov 02 23:41:22.624242 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:22.624200 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e316f95-0005-44bd-a11b-8b04b3e32297-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e316f95-0005-44bd-a11b-8b04b3e32297-metrics-certs\") pod \"5e316f95-0005-44bd-a11b-8b04b3e32297\" (UID: \"5e316f95-0005-44bd-a11b-8b04b3e32297\") " Nov 02 23:41:22.624735 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:22.624715 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e316f95-0005-44bd-a11b-8b04b3e32297-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e316f95-0005-44bd-a11b-8b04b3e32297-openshift-service-ca\") pod \"5e316f95-0005-44bd-a11b-8b04b3e32297\" (UID: \"5e316f95-0005-44bd-a11b-8b04b3e32297\") " Nov 02 23:41:22.625741 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:22.625605 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e316f95-0005-44bd-a11b-8b04b3e32297-openshift-service-ca" pod "5e316f95-0005-44bd-a11b-8b04b3e32297" (UID: "5e316f95-0005-44bd-a11b-8b04b3e32297"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:41:22.627230 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:22.627089 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e316f95-0005-44bd-a11b-8b04b3e32297-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:41:22.630295 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:22.630272 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e316f95-0005-44bd-a11b-8b04b3e32297-kube-api-access-hq8dv" pod "5e316f95-0005-44bd-a11b-8b04b3e32297" (UID: "5e316f95-0005-44bd-a11b-8b04b3e32297"). InnerVolumeSpecName "kube-api-access-hq8dv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:41:22.632087 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:22.632049 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e316f95-0005-44bd-a11b-8b04b3e32297-metrics-certs" pod "5e316f95-0005-44bd-a11b-8b04b3e32297" (UID: "5e316f95-0005-44bd-a11b-8b04b3e32297"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:41:22.632683 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:22.632662 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e316f95-0005-44bd-a11b-8b04b3e32297-maas-api-tls" pod "5e316f95-0005-44bd-a11b-8b04b3e32297" (UID: "5e316f95-0005-44bd-a11b-8b04b3e32297"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:41:22.765705 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:22.762673 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e316f95-0005-44bd-a11b-8b04b3e32297-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:41:22.765705 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:22.762715 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq8dv\" (UniqueName: \"kubernetes.io/projected/5e316f95-0005-44bd-a11b-8b04b3e32297-kube-api-access-hq8dv\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:41:22.765705 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:22.762731 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e316f95-0005-44bd-a11b-8b04b3e32297-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:41:23.148990 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:23.142444 5076 generic.go:356] "Generic (PLEG): container finished" podID="5e316f95-0005-44bd-a11b-8b04b3e32297" containerID="ca42ef19bab30ca0be9870bf41c21b8ace331ebeb55ecfca5db6c87a26b37dc6" exitCode=0 Nov 02 23:41:23.148990 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:23.142751 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4" event={"ID":"5e316f95-0005-44bd-a11b-8b04b3e32297","Type":"ContainerDied","Data":"ca42ef19bab30ca0be9870bf41c21b8ace331ebeb55ecfca5db6c87a26b37dc6"} Nov 02 23:41:23.148990 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:23.142830 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4" event={"ID":"5e316f95-0005-44bd-a11b-8b04b3e32297","Type":"ContainerDied","Data":"f2724fc1e872aec001ad2ff5b204f5e6919a9704dafda1a48a64a743f7eb38c6"} Nov 02 23:41:23.148990 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:23.142864 5076 scope.go:122] "RemoveContainer" containerID="ca42ef19bab30ca0be9870bf41c21b8ace331ebeb55ecfca5db6c87a26b37dc6" Nov 02 23:41:23.148990 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:23.143055 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4" Nov 02 23:41:23.264500 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:23.264469 5076 scope.go:122] "RemoveContainer" containerID="ca42ef19bab30ca0be9870bf41c21b8ace331ebeb55ecfca5db6c87a26b37dc6" Nov 02 23:41:23.265101 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:41:23.264877 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca42ef19bab30ca0be9870bf41c21b8ace331ebeb55ecfca5db6c87a26b37dc6\": container with ID starting with ca42ef19bab30ca0be9870bf41c21b8ace331ebeb55ecfca5db6c87a26b37dc6 not found: ID does not exist" containerID="ca42ef19bab30ca0be9870bf41c21b8ace331ebeb55ecfca5db6c87a26b37dc6" Nov 02 23:41:23.265101 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:23.264950 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca42ef19bab30ca0be9870bf41c21b8ace331ebeb55ecfca5db6c87a26b37dc6"} err="failed to get container status \"ca42ef19bab30ca0be9870bf41c21b8ace331ebeb55ecfca5db6c87a26b37dc6\": rpc error: code = NotFound desc = could not find container \"ca42ef19bab30ca0be9870bf41c21b8ace331ebeb55ecfca5db6c87a26b37dc6\": container with ID starting with ca42ef19bab30ca0be9870bf41c21b8ace331ebeb55ecfca5db6c87a26b37dc6 not found: ID does not exist" Nov 02 23:41:23.369954 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:23.369917 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4"] Nov 02 23:41:23.416952 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:23.416848 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-9672a1-564b4d55-c74b4"] Nov 02 23:41:25.148989 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:25.148955 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e316f95-0005-44bd-a11b-8b04b3e32297" path="/var/lib/kubelet/pods/5e316f95-0005-44bd-a11b-8b04b3e32297/volumes" Nov 02 23:41:28.583219 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:28.583182 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz"] Nov 02 23:41:28.584031 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:28.584010 5076 cpu_manager.go:414] "containerMap: removing container" podUID="5e316f95-0005-44bd-a11b-8b04b3e32297" containerName="maas-api" Nov 02 23:41:28.584143 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:28.584032 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e316f95-0005-44bd-a11b-8b04b3e32297" containerName="maas-api" Nov 02 23:41:28.584201 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:28.584167 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e316f95-0005-44bd-a11b-8b04b3e32297" containerName="maas-api" Nov 02 23:41:28.589173 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:28.589151 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz" Nov 02 23:41:28.641064 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:28.641027 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz"] Nov 02 23:41:28.644733 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:28.644698 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0ae7fef-665b-4ac5-9028-9214d1a9fe47-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz\" (UID: \"e0ae7fef-665b-4ac5-9028-9214d1a9fe47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz" Nov 02 23:41:28.644926 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:28.644869 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlf49\" (UniqueName: \"kubernetes.io/projected/e0ae7fef-665b-4ac5-9028-9214d1a9fe47-kube-api-access-vlf49\") pod \"maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz\" (UID: \"e0ae7fef-665b-4ac5-9028-9214d1a9fe47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz" Nov 02 23:41:28.645076 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:28.645051 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0ae7fef-665b-4ac5-9028-9214d1a9fe47-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz\" (UID: \"e0ae7fef-665b-4ac5-9028-9214d1a9fe47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz" Nov 02 23:41:28.645186 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:28.645168 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0ae7fef-665b-4ac5-9028-9214d1a9fe47-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz\" (UID: \"e0ae7fef-665b-4ac5-9028-9214d1a9fe47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz" Nov 02 23:41:28.684795 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:28.684748 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-7ca60f-serving-cert\"" Nov 02 23:41:28.746314 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:28.746277 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0ae7fef-665b-4ac5-9028-9214d1a9fe47-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz\" (UID: \"e0ae7fef-665b-4ac5-9028-9214d1a9fe47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz" Nov 02 23:41:28.746508 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:28.746367 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlf49\" (UniqueName: \"kubernetes.io/projected/e0ae7fef-665b-4ac5-9028-9214d1a9fe47-kube-api-access-vlf49\") pod \"maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz\" (UID: \"e0ae7fef-665b-4ac5-9028-9214d1a9fe47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz" Nov 02 23:41:28.746508 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:28.746444 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0ae7fef-665b-4ac5-9028-9214d1a9fe47-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz\" (UID: \"e0ae7fef-665b-4ac5-9028-9214d1a9fe47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz" Nov 02 23:41:28.746651 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:28.746513 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0ae7fef-665b-4ac5-9028-9214d1a9fe47-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz\" (UID: \"e0ae7fef-665b-4ac5-9028-9214d1a9fe47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz" Nov 02 23:41:28.746770 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:41:28.746747 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ca60f-serving-cert: secret "maas-api-e2e-shared-b-w3-7ca60f-serving-cert" not found Nov 02 23:41:28.746843 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:41:28.746828 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0ae7fef-665b-4ac5-9028-9214d1a9fe47-maas-api-tls podName:e0ae7fef-665b-4ac5-9028-9214d1a9fe47 nodeName:}" failed. No retries permitted until 2026-11-02 23:41:29.246805621 +0000 UTC m=+2938.215337399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e0ae7fef-665b-4ac5-9028-9214d1a9fe47-maas-api-tls") pod "maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz" (UID: "e0ae7fef-665b-4ac5-9028-9214d1a9fe47") : secret "maas-api-e2e-shared-b-w3-7ca60f-serving-cert" not found Nov 02 23:41:28.747541 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:28.747509 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0ae7fef-665b-4ac5-9028-9214d1a9fe47-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz\" (UID: \"e0ae7fef-665b-4ac5-9028-9214d1a9fe47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz" Nov 02 23:41:28.749211 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:28.749188 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0ae7fef-665b-4ac5-9028-9214d1a9fe47-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz\" (UID: \"e0ae7fef-665b-4ac5-9028-9214d1a9fe47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz" Nov 02 23:41:28.803644 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:28.803607 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlf49\" (UniqueName: \"kubernetes.io/projected/e0ae7fef-665b-4ac5-9028-9214d1a9fe47-kube-api-access-vlf49\") pod \"maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz\" (UID: \"e0ae7fef-665b-4ac5-9028-9214d1a9fe47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz" Nov 02 23:41:29.256019 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:29.255979 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0ae7fef-665b-4ac5-9028-9214d1a9fe47-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz\" (UID: \"e0ae7fef-665b-4ac5-9028-9214d1a9fe47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz" Nov 02 23:41:29.258504 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:29.258479 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0ae7fef-665b-4ac5-9028-9214d1a9fe47-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz\" (UID: \"e0ae7fef-665b-4ac5-9028-9214d1a9fe47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz" Nov 02 23:41:29.503232 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:29.503201 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz" Nov 02 23:41:29.874427 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:29.874397 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz"] Nov 02 23:41:29.875260 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:41:29.875214 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ae7fef_665b_4ac5_9028_9214d1a9fe47.slice/crio-b07473830ddf0c60f6252c35df81647c1e18f50273e68155a2702a38e492e775 WatchSource:0}: Error finding container b07473830ddf0c60f6252c35df81647c1e18f50273e68155a2702a38e492e775: Status 404 returned error can't find the container with id b07473830ddf0c60f6252c35df81647c1e18f50273e68155a2702a38e492e775 Nov 02 23:41:30.186858 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:30.186821 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz" event={"ID":"e0ae7fef-665b-4ac5-9028-9214d1a9fe47","Type":"ContainerStarted","Data":"b07473830ddf0c60f6252c35df81647c1e18f50273e68155a2702a38e492e775"} Nov 02 23:41:31.376042 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:31.375995 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz" event={"ID":"e0ae7fef-665b-4ac5-9028-9214d1a9fe47","Type":"ContainerStarted","Data":"9a33710e99c68bbc052f578515aaf8227f6f81ce0cf33e0d69206ee9ceeca769"} Nov 02 23:41:31.376673 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:31.376642 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz" Nov 02 23:41:31.530924 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:31.525700 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz" podStartSLOduration=2.661637967 podStartE2EDuration="3.525678491s" totalImagesPullingTime="864.040524ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:41:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:41:31.524250507 +0000 UTC m=+2940.492782291" watchObservedRunningTime="2026-11-02 23:41:31.525678491 +0000 UTC m=+2940.494210276" Nov 02 23:41:33.513287 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:33.513250 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x"] Nov 02 23:41:33.513767 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:33.513514 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" podUID="3fd88117-cf15-4806-ade9-c17982846d7d" containerName="istio-proxy" containerID="cri-o://252dc8ada1fc2473291145ed0d3ec3dd164bcd02b51b284eaebda2ba36678974" gracePeriod=30 Nov 02 23:41:34.346999 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:34.346964 5076 patch_prober.go:29] interesting pod/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:41:34.347526 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:34.347365 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" podUID="3fd88117-cf15-4806-ade9-c17982846d7d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:41:37.387660 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:37.387629 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz" Nov 02 23:41:38.783484 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:38.783410 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" Nov 02 23:41:38.911005 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:38.910964 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fd88117-cf15-4806-ade9-c17982846d7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fd88117-cf15-4806-ade9-c17982846d7d-istiod-ca-cert\") pod \"3fd88117-cf15-4806-ade9-c17982846d7d\" (UID: \"3fd88117-cf15-4806-ade9-c17982846d7d\") " Nov 02 23:41:38.911005 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:38.911011 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3fd88117-cf15-4806-ade9-c17982846d7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fd88117-cf15-4806-ade9-c17982846d7d-istio-podinfo\") pod \"3fd88117-cf15-4806-ade9-c17982846d7d\" (UID: \"3fd88117-cf15-4806-ade9-c17982846d7d\") " Nov 02 23:41:38.911277 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:38.911192 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fd88117-cf15-4806-ade9-c17982846d7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fd88117-cf15-4806-ade9-c17982846d7d-istio-data\") pod \"3fd88117-cf15-4806-ade9-c17982846d7d\" (UID: \"3fd88117-cf15-4806-ade9-c17982846d7d\") " Nov 02 23:41:38.911341 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:38.911324 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fd88117-cf15-4806-ade9-c17982846d7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd88117-cf15-4806-ade9-c17982846d7d-credential-socket\") pod \"3fd88117-cf15-4806-ade9-c17982846d7d\" (UID: \"3fd88117-cf15-4806-ade9-c17982846d7d\") " Nov 02 23:41:38.911423 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:38.911404 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fd88117-cf15-4806-ade9-c17982846d7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fd88117-cf15-4806-ade9-c17982846d7d-istio-envoy\") pod \"3fd88117-cf15-4806-ade9-c17982846d7d\" (UID: \"3fd88117-cf15-4806-ade9-c17982846d7d\") " Nov 02 23:41:38.911485 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:38.911433 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fd88117-cf15-4806-ade9-c17982846d7d-istio-data" pod "3fd88117-cf15-4806-ade9-c17982846d7d" (UID: "3fd88117-cf15-4806-ade9-c17982846d7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:41:38.911485 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:38.911455 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fd88117-cf15-4806-ade9-c17982846d7d-istio-token\" (UniqueName: \"kubernetes.io/projected/3fd88117-cf15-4806-ade9-c17982846d7d-istio-token\") pod \"3fd88117-cf15-4806-ade9-c17982846d7d\" (UID: \"3fd88117-cf15-4806-ade9-c17982846d7d\") " Nov 02 23:41:38.911485 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:38.911464 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fd88117-cf15-4806-ade9-c17982846d7d-istiod-ca-cert" pod "3fd88117-cf15-4806-ade9-c17982846d7d" (UID: "3fd88117-cf15-4806-ade9-c17982846d7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:41:38.911877 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:38.911493 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fd88117-cf15-4806-ade9-c17982846d7d-kube-api-access-8thrr\" (UniqueName: \"kubernetes.io/projected/3fd88117-cf15-4806-ade9-c17982846d7d-kube-api-access-8thrr\") pod \"3fd88117-cf15-4806-ade9-c17982846d7d\" (UID: \"3fd88117-cf15-4806-ade9-c17982846d7d\") " Nov 02 23:41:38.911877 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:38.911541 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fd88117-cf15-4806-ade9-c17982846d7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fd88117-cf15-4806-ade9-c17982846d7d-workload-certs\") pod \"3fd88117-cf15-4806-ade9-c17982846d7d\" (UID: \"3fd88117-cf15-4806-ade9-c17982846d7d\") " Nov 02 23:41:38.911877 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:38.911557 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fd88117-cf15-4806-ade9-c17982846d7d-credential-socket" pod "3fd88117-cf15-4806-ade9-c17982846d7d" (UID: "3fd88117-cf15-4806-ade9-c17982846d7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:41:38.911877 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:38.911581 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fd88117-cf15-4806-ade9-c17982846d7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd88117-cf15-4806-ade9-c17982846d7d-workload-socket\") pod \"3fd88117-cf15-4806-ade9-c17982846d7d\" (UID: \"3fd88117-cf15-4806-ade9-c17982846d7d\") " Nov 02 23:41:38.911877 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:38.911772 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fd88117-cf15-4806-ade9-c17982846d7d-workload-certs" pod "3fd88117-cf15-4806-ade9-c17982846d7d" (UID: "3fd88117-cf15-4806-ade9-c17982846d7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:41:38.912262 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:38.911915 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fd88117-cf15-4806-ade9-c17982846d7d-workload-socket" pod "3fd88117-cf15-4806-ade9-c17982846d7d" (UID: "3fd88117-cf15-4806-ade9-c17982846d7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:41:38.912262 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:38.912209 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fd88117-cf15-4806-ade9-c17982846d7d-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:41:38.912262 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:38.912226 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd88117-cf15-4806-ade9-c17982846d7d-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:41:38.912262 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:38.912236 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fd88117-cf15-4806-ade9-c17982846d7d-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:41:38.912262 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:38.912248 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fd88117-cf15-4806-ade9-c17982846d7d-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:41:38.912262 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:38.912262 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd88117-cf15-4806-ade9-c17982846d7d-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:41:38.913655 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:38.913628 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fd88117-cf15-4806-ade9-c17982846d7d-kube-api-access-8thrr" pod "3fd88117-cf15-4806-ade9-c17982846d7d" (UID: "3fd88117-cf15-4806-ade9-c17982846d7d"). InnerVolumeSpecName "kube-api-access-8thrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:41:38.913819 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:38.913800 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fd88117-cf15-4806-ade9-c17982846d7d-istio-token" pod "3fd88117-cf15-4806-ade9-c17982846d7d" (UID: "3fd88117-cf15-4806-ade9-c17982846d7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:41:38.913928 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:38.913813 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fd88117-cf15-4806-ade9-c17982846d7d-istio-podinfo" pod "3fd88117-cf15-4806-ade9-c17982846d7d" (UID: "3fd88117-cf15-4806-ade9-c17982846d7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:41:38.913928 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:38.913800 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fd88117-cf15-4806-ade9-c17982846d7d-istio-envoy" pod "3fd88117-cf15-4806-ade9-c17982846d7d" (UID: "3fd88117-cf15-4806-ade9-c17982846d7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:41:39.014070 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:39.014032 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fd88117-cf15-4806-ade9-c17982846d7d-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:41:39.014070 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:39.014067 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fd88117-cf15-4806-ade9-c17982846d7d-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:41:39.014280 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:39.014081 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8thrr\" (UniqueName: \"kubernetes.io/projected/3fd88117-cf15-4806-ade9-c17982846d7d-kube-api-access-8thrr\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:41:39.014280 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:39.014095 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fd88117-cf15-4806-ade9-c17982846d7d-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:41:39.427072 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:39.426970 5076 generic.go:356] "Generic (PLEG): container finished" podID="3fd88117-cf15-4806-ade9-c17982846d7d" containerID="252dc8ada1fc2473291145ed0d3ec3dd164bcd02b51b284eaebda2ba36678974" exitCode=0 Nov 02 23:41:39.427072 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:39.427020 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" event={"ID":"3fd88117-cf15-4806-ade9-c17982846d7d","Type":"ContainerDied","Data":"252dc8ada1fc2473291145ed0d3ec3dd164bcd02b51b284eaebda2ba36678974"} Nov 02 23:41:39.427072 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:39.427071 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" event={"ID":"3fd88117-cf15-4806-ade9-c17982846d7d","Type":"ContainerDied","Data":"dee5ca8dbf60f79b7ac87479cfb742018ef3037261c7ac8853787ec624999bc8"} Nov 02 23:41:39.427354 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:39.427027 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x" Nov 02 23:41:39.427354 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:39.427100 5076 scope.go:122] "RemoveContainer" containerID="252dc8ada1fc2473291145ed0d3ec3dd164bcd02b51b284eaebda2ba36678974" Nov 02 23:41:39.436824 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:39.436799 5076 scope.go:122] "RemoveContainer" containerID="b806b7dfdb2b05430ae52c431243adad3d9c999fc29dac4f7e47874c6a5db367" Nov 02 23:41:39.445791 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:39.445767 5076 scope.go:122] "RemoveContainer" containerID="252dc8ada1fc2473291145ed0d3ec3dd164bcd02b51b284eaebda2ba36678974" Nov 02 23:41:39.446093 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:41:39.446072 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"252dc8ada1fc2473291145ed0d3ec3dd164bcd02b51b284eaebda2ba36678974\": container with ID starting with 252dc8ada1fc2473291145ed0d3ec3dd164bcd02b51b284eaebda2ba36678974 not found: ID does not exist" containerID="252dc8ada1fc2473291145ed0d3ec3dd164bcd02b51b284eaebda2ba36678974" Nov 02 23:41:39.446150 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:39.446103 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"252dc8ada1fc2473291145ed0d3ec3dd164bcd02b51b284eaebda2ba36678974"} err="failed to get container status \"252dc8ada1fc2473291145ed0d3ec3dd164bcd02b51b284eaebda2ba36678974\": rpc error: code = NotFound desc = could not find container \"252dc8ada1fc2473291145ed0d3ec3dd164bcd02b51b284eaebda2ba36678974\": container with ID starting with 252dc8ada1fc2473291145ed0d3ec3dd164bcd02b51b284eaebda2ba36678974 not found: ID does not exist" Nov 02 23:41:39.446150 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:39.446128 5076 scope.go:122] "RemoveContainer" containerID="b806b7dfdb2b05430ae52c431243adad3d9c999fc29dac4f7e47874c6a5db367" Nov 02 23:41:39.446362 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:41:39.446346 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b806b7dfdb2b05430ae52c431243adad3d9c999fc29dac4f7e47874c6a5db367\": container with ID starting with b806b7dfdb2b05430ae52c431243adad3d9c999fc29dac4f7e47874c6a5db367 not found: ID does not exist" containerID="b806b7dfdb2b05430ae52c431243adad3d9c999fc29dac4f7e47874c6a5db367" Nov 02 23:41:39.446400 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:39.446367 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b806b7dfdb2b05430ae52c431243adad3d9c999fc29dac4f7e47874c6a5db367"} err="failed to get container status \"b806b7dfdb2b05430ae52c431243adad3d9c999fc29dac4f7e47874c6a5db367\": rpc error: code = NotFound desc = could not find container \"b806b7dfdb2b05430ae52c431243adad3d9c999fc29dac4f7e47874c6a5db367\": container with ID starting with b806b7dfdb2b05430ae52c431243adad3d9c999fc29dac4f7e47874c6a5db367 not found: ID does not exist" Nov 02 23:41:39.586318 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:39.586286 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x"] Nov 02 23:41:39.633553 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:39.633512 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-9672a1-openshift-default-5b98fc4df-d777x"] Nov 02 23:41:39.884388 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:41:39.884342 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:41:41.149416 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:41:41.149385 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fd88117-cf15-4806-ade9-c17982846d7d" path="/var/lib/kubelet/pods/3fd88117-cf15-4806-ade9-c17982846d7d/volumes" Nov 02 23:42:07.676212 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:42:07.676175 5076 generic.go:356] "Generic (PLEG): container finished" podID="6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e" containerID="ebc686f9abf6989bad70a068d8bc3eade1b424a6ee79d59586fb29d8850cb6df" exitCode=0 Nov 02 23:42:07.676655 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:42:07.676247 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" event={"ID":"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e","Type":"ContainerDied","Data":"ebc686f9abf6989bad70a068d8bc3eade1b424a6ee79d59586fb29d8850cb6df"} Nov 02 23:42:07.676702 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:42:07.676667 5076 scope.go:122] "RemoveContainer" containerID="ebc686f9abf6989bad70a068d8bc3eade1b424a6ee79d59586fb29d8850cb6df" Nov 02 23:42:07.678242 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:42:07.678211 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:42:07.678353 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:42:07.678299 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:42:07.678353 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:42:07.678346 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:42:08.684145 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:42:08.684110 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" event={"ID":"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e","Type":"ContainerStarted","Data":"3926e367f244454e6746785c2f8fa3b7c4685c5d19139ac881082d875b310d7d"} Nov 02 23:42:09.243807 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:42:09.243770 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" Nov 02 23:42:09.244064 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:42:09.243948 5076 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 02 23:42:09.244064 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:42:09.243997 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" podUID="6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 02 23:42:09.901949 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:42:09.901877 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:42:10.243307 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:42:10.243270 5076 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 02 23:42:10.243494 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:42:10.243339 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" podUID="6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 02 23:42:11.243475 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:42:11.243434 5076 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 02 23:42:11.244008 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:42:11.243501 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" podUID="6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 02 23:42:12.243341 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:42:12.243309 5076 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 02 23:42:12.243540 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:42:12.243370 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" podUID="6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 02 23:42:13.243469 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:42:13.243427 5076 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 02 23:42:13.243692 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:42:13.243511 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" podUID="6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 02 23:42:14.249397 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:42:14.249362 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" Nov 02 23:42:14.250345 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:42:14.249854 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" Nov 02 23:42:14.250689 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:42:14.250665 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" Nov 02 23:42:31.886537 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:42:31.886509 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 02 23:42:31.919238 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:42:31.919211 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 02 23:42:39.886963 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:42:39.886924 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:43:02.845376 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:02.845289 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8"] Nov 02 23:43:02.845831 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:02.845558 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" podUID="974b990b-722c-427a-9328-6f78b23b63d3" containerName="istio-proxy" containerID="cri-o://e35c61b916989d98518ac882c21fe90487e47ac28b122ac22194eb760aece8fd" gracePeriod=30 Nov 02 23:43:08.113936 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.113851 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" Nov 02 23:43:08.148520 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.148491 5076 generic.go:356] "Generic (PLEG): container finished" podID="974b990b-722c-427a-9328-6f78b23b63d3" containerID="e35c61b916989d98518ac882c21fe90487e47ac28b122ac22194eb760aece8fd" exitCode=0 Nov 02 23:43:08.148688 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.148537 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" Nov 02 23:43:08.148688 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.148560 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" event={"ID":"974b990b-722c-427a-9328-6f78b23b63d3","Type":"ContainerDied","Data":"e35c61b916989d98518ac882c21fe90487e47ac28b122ac22194eb760aece8fd"} Nov 02 23:43:08.148688 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.148608 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8" event={"ID":"974b990b-722c-427a-9328-6f78b23b63d3","Type":"ContainerDied","Data":"ad35c87ed65b2447c01124b3c7276d89d78bf562185b08c320203e6ec014f837"} Nov 02 23:43:08.148688 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.148628 5076 scope.go:122] "RemoveContainer" containerID="e35c61b916989d98518ac882c21fe90487e47ac28b122ac22194eb760aece8fd" Nov 02 23:43:08.149779 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.149759 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/974b990b-722c-427a-9328-6f78b23b63d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/974b990b-722c-427a-9328-6f78b23b63d3-istio-data\") pod \"974b990b-722c-427a-9328-6f78b23b63d3\" (UID: \"974b990b-722c-427a-9328-6f78b23b63d3\") " Nov 02 23:43:08.149870 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.149822 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/974b990b-722c-427a-9328-6f78b23b63d3-kube-api-access-mtn24\" (UniqueName: \"kubernetes.io/projected/974b990b-722c-427a-9328-6f78b23b63d3-kube-api-access-mtn24\") pod \"974b990b-722c-427a-9328-6f78b23b63d3\" (UID: \"974b990b-722c-427a-9328-6f78b23b63d3\") " Nov 02 23:43:08.149870 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.149848 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/974b990b-722c-427a-9328-6f78b23b63d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/974b990b-722c-427a-9328-6f78b23b63d3-istio-podinfo\") pod \"974b990b-722c-427a-9328-6f78b23b63d3\" (UID: \"974b990b-722c-427a-9328-6f78b23b63d3\") " Nov 02 23:43:08.150086 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.149870 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/974b990b-722c-427a-9328-6f78b23b63d3-istio-token\" (UniqueName: \"kubernetes.io/projected/974b990b-722c-427a-9328-6f78b23b63d3-istio-token\") pod \"974b990b-722c-427a-9328-6f78b23b63d3\" (UID: \"974b990b-722c-427a-9328-6f78b23b63d3\") " Nov 02 23:43:08.150086 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.149932 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/974b990b-722c-427a-9328-6f78b23b63d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/974b990b-722c-427a-9328-6f78b23b63d3-istio-envoy\") pod \"974b990b-722c-427a-9328-6f78b23b63d3\" (UID: \"974b990b-722c-427a-9328-6f78b23b63d3\") " Nov 02 23:43:08.150086 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.150023 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/974b990b-722c-427a-9328-6f78b23b63d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/974b990b-722c-427a-9328-6f78b23b63d3-credential-socket\") pod \"974b990b-722c-427a-9328-6f78b23b63d3\" (UID: \"974b990b-722c-427a-9328-6f78b23b63d3\") " Nov 02 23:43:08.150086 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.150055 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/974b990b-722c-427a-9328-6f78b23b63d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/974b990b-722c-427a-9328-6f78b23b63d3-workload-certs\") pod \"974b990b-722c-427a-9328-6f78b23b63d3\" (UID: \"974b990b-722c-427a-9328-6f78b23b63d3\") " Nov 02 23:43:08.150086 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.150079 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/974b990b-722c-427a-9328-6f78b23b63d3-istio-data" pod "974b990b-722c-427a-9328-6f78b23b63d3" (UID: "974b990b-722c-427a-9328-6f78b23b63d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:08.150382 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.150181 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/974b990b-722c-427a-9328-6f78b23b63d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/974b990b-722c-427a-9328-6f78b23b63d3-istiod-ca-cert\") pod \"974b990b-722c-427a-9328-6f78b23b63d3\" (UID: \"974b990b-722c-427a-9328-6f78b23b63d3\") " Nov 02 23:43:08.150382 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.150242 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/974b990b-722c-427a-9328-6f78b23b63d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/974b990b-722c-427a-9328-6f78b23b63d3-workload-socket\") pod \"974b990b-722c-427a-9328-6f78b23b63d3\" (UID: \"974b990b-722c-427a-9328-6f78b23b63d3\") " Nov 02 23:43:08.150382 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.150293 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/974b990b-722c-427a-9328-6f78b23b63d3-credential-socket" pod "974b990b-722c-427a-9328-6f78b23b63d3" (UID: "974b990b-722c-427a-9328-6f78b23b63d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:08.150554 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.150479 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/974b990b-722c-427a-9328-6f78b23b63d3-workload-certs" pod "974b990b-722c-427a-9328-6f78b23b63d3" (UID: "974b990b-722c-427a-9328-6f78b23b63d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:08.150744 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.150716 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/974b990b-722c-427a-9328-6f78b23b63d3-workload-socket" pod "974b990b-722c-427a-9328-6f78b23b63d3" (UID: "974b990b-722c-427a-9328-6f78b23b63d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:08.150818 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.150756 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/974b990b-722c-427a-9328-6f78b23b63d3-istiod-ca-cert" pod "974b990b-722c-427a-9328-6f78b23b63d3" (UID: "974b990b-722c-427a-9328-6f78b23b63d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:43:08.150923 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.150885 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/974b990b-722c-427a-9328-6f78b23b63d3-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:08.151101 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.151079 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/974b990b-722c-427a-9328-6f78b23b63d3-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:08.151209 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.151104 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/974b990b-722c-427a-9328-6f78b23b63d3-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:08.151209 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.151119 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/974b990b-722c-427a-9328-6f78b23b63d3-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:08.151209 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.151135 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/974b990b-722c-427a-9328-6f78b23b63d3-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:08.152326 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.152305 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/974b990b-722c-427a-9328-6f78b23b63d3-istio-token" pod "974b990b-722c-427a-9328-6f78b23b63d3" (UID: "974b990b-722c-427a-9328-6f78b23b63d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:43:08.152439 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.152343 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/974b990b-722c-427a-9328-6f78b23b63d3-kube-api-access-mtn24" pod "974b990b-722c-427a-9328-6f78b23b63d3" (UID: "974b990b-722c-427a-9328-6f78b23b63d3"). InnerVolumeSpecName "kube-api-access-mtn24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:43:08.152549 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.152532 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/974b990b-722c-427a-9328-6f78b23b63d3-istio-podinfo" pod "974b990b-722c-427a-9328-6f78b23b63d3" (UID: "974b990b-722c-427a-9328-6f78b23b63d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:43:08.152788 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.152763 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/974b990b-722c-427a-9328-6f78b23b63d3-istio-envoy" pod "974b990b-722c-427a-9328-6f78b23b63d3" (UID: "974b990b-722c-427a-9328-6f78b23b63d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:08.179063 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.179037 5076 scope.go:122] "RemoveContainer" containerID="9517d0775eca1f38f2203a75a86b5a82bd7b5b05f6365c590a416fb3929a2e40" Nov 02 23:43:08.189878 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.189857 5076 scope.go:122] "RemoveContainer" containerID="e35c61b916989d98518ac882c21fe90487e47ac28b122ac22194eb760aece8fd" Nov 02 23:43:08.190193 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:43:08.190174 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e35c61b916989d98518ac882c21fe90487e47ac28b122ac22194eb760aece8fd\": container with ID starting with e35c61b916989d98518ac882c21fe90487e47ac28b122ac22194eb760aece8fd not found: ID does not exist" containerID="e35c61b916989d98518ac882c21fe90487e47ac28b122ac22194eb760aece8fd" Nov 02 23:43:08.190242 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.190202 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e35c61b916989d98518ac882c21fe90487e47ac28b122ac22194eb760aece8fd"} err="failed to get container status \"e35c61b916989d98518ac882c21fe90487e47ac28b122ac22194eb760aece8fd\": rpc error: code = NotFound desc = could not find container \"e35c61b916989d98518ac882c21fe90487e47ac28b122ac22194eb760aece8fd\": container with ID starting with e35c61b916989d98518ac882c21fe90487e47ac28b122ac22194eb760aece8fd not found: ID does not exist" Nov 02 23:43:08.190242 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.190222 5076 scope.go:122] "RemoveContainer" containerID="9517d0775eca1f38f2203a75a86b5a82bd7b5b05f6365c590a416fb3929a2e40" Nov 02 23:43:08.190497 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:43:08.190474 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9517d0775eca1f38f2203a75a86b5a82bd7b5b05f6365c590a416fb3929a2e40\": container with ID starting with 9517d0775eca1f38f2203a75a86b5a82bd7b5b05f6365c590a416fb3929a2e40 not found: ID does not exist" containerID="9517d0775eca1f38f2203a75a86b5a82bd7b5b05f6365c590a416fb3929a2e40" Nov 02 23:43:08.190540 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.190504 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9517d0775eca1f38f2203a75a86b5a82bd7b5b05f6365c590a416fb3929a2e40"} err="failed to get container status \"9517d0775eca1f38f2203a75a86b5a82bd7b5b05f6365c590a416fb3929a2e40\": rpc error: code = NotFound desc = could not find container \"9517d0775eca1f38f2203a75a86b5a82bd7b5b05f6365c590a416fb3929a2e40\": container with ID starting with 9517d0775eca1f38f2203a75a86b5a82bd7b5b05f6365c590a416fb3929a2e40 not found: ID does not exist" Nov 02 23:43:08.252353 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.252321 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtn24\" (UniqueName: \"kubernetes.io/projected/974b990b-722c-427a-9328-6f78b23b63d3-kube-api-access-mtn24\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:08.252353 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.252348 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/974b990b-722c-427a-9328-6f78b23b63d3-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:08.252353 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.252358 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/974b990b-722c-427a-9328-6f78b23b63d3-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:08.252576 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.252368 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/974b990b-722c-427a-9328-6f78b23b63d3-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:08.564536 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.564496 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8"] Nov 02 23:43:08.570198 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:08.570174 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-b1be95-openshift-default-54867cc477-bzmf8"] Nov 02 23:43:09.156775 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:09.156741 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="974b990b-722c-427a-9328-6f78b23b63d3" path="/var/lib/kubelet/pods/974b990b-722c-427a-9328-6f78b23b63d3/volumes" Nov 02 23:43:09.884213 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:43:09.884176 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:43:11.657699 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.657665 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w"] Nov 02 23:43:11.658299 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.658284 5076 cpu_manager.go:414] "containerMap: removing container" podUID="974b990b-722c-427a-9328-6f78b23b63d3" containerName="istio-proxy" Nov 02 23:43:11.658346 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.658300 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="974b990b-722c-427a-9328-6f78b23b63d3" containerName="istio-proxy" Nov 02 23:43:11.658382 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.658345 5076 cpu_manager.go:414] "containerMap: removing container" podUID="3fd88117-cf15-4806-ade9-c17982846d7d" containerName="istio-proxy" Nov 02 23:43:11.658382 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.658351 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fd88117-cf15-4806-ade9-c17982846d7d" containerName="istio-proxy" Nov 02 23:43:11.658382 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.658363 5076 cpu_manager.go:414] "containerMap: removing container" podUID="3fd88117-cf15-4806-ade9-c17982846d7d" containerName="istio-proxy" Nov 02 23:43:11.658382 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.658368 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fd88117-cf15-4806-ade9-c17982846d7d" containerName="istio-proxy" Nov 02 23:43:11.658506 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.658451 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="974b990b-722c-427a-9328-6f78b23b63d3" containerName="istio-proxy" Nov 02 23:43:11.658506 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.658465 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="974b990b-722c-427a-9328-6f78b23b63d3" containerName="istio-proxy" Nov 02 23:43:11.658506 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.658472 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fd88117-cf15-4806-ade9-c17982846d7d" containerName="istio-proxy" Nov 02 23:43:11.658506 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.658479 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fd88117-cf15-4806-ade9-c17982846d7d" containerName="istio-proxy" Nov 02 23:43:11.663564 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.663539 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" Nov 02 23:43:11.709229 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.704610 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae04339e-7f59-4edd-862a-aa88108d59bb-istio-data\") pod \"e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w\" (UID: \"ae04339e-7f59-4edd-862a-aa88108d59bb\") " pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" Nov 02 23:43:11.709229 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.704674 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae04339e-7f59-4edd-862a-aa88108d59bb-istio-podinfo\") pod \"e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w\" (UID: \"ae04339e-7f59-4edd-862a-aa88108d59bb\") " pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" Nov 02 23:43:11.709229 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.704720 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae04339e-7f59-4edd-862a-aa88108d59bb-istio-token\") pod \"e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w\" (UID: \"ae04339e-7f59-4edd-862a-aa88108d59bb\") " pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" Nov 02 23:43:11.709229 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.704761 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae04339e-7f59-4edd-862a-aa88108d59bb-workload-socket\") pod \"e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w\" (UID: \"ae04339e-7f59-4edd-862a-aa88108d59bb\") " pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" Nov 02 23:43:11.709229 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.704824 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae04339e-7f59-4edd-862a-aa88108d59bb-workload-certs\") pod \"e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w\" (UID: \"ae04339e-7f59-4edd-862a-aa88108d59bb\") " pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" Nov 02 23:43:11.709229 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.705011 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae04339e-7f59-4edd-862a-aa88108d59bb-credential-socket\") pod \"e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w\" (UID: \"ae04339e-7f59-4edd-862a-aa88108d59bb\") " pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" Nov 02 23:43:11.709229 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.705061 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlj2x\" (UniqueName: \"kubernetes.io/projected/ae04339e-7f59-4edd-862a-aa88108d59bb-kube-api-access-qlj2x\") pod \"e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w\" (UID: \"ae04339e-7f59-4edd-862a-aa88108d59bb\") " pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" Nov 02 23:43:11.709229 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.705097 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae04339e-7f59-4edd-862a-aa88108d59bb-istiod-ca-cert\") pod \"e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w\" (UID: \"ae04339e-7f59-4edd-862a-aa88108d59bb\") " pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" Nov 02 23:43:11.709229 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.705150 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae04339e-7f59-4edd-862a-aa88108d59bb-istio-envoy\") pod \"e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w\" (UID: \"ae04339e-7f59-4edd-862a-aa88108d59bb\") " pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" Nov 02 23:43:11.714364 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.714322 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-52dd59-openshift-default-dockercfg-hfxrc\"" Nov 02 23:43:11.770669 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.770631 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w"] Nov 02 23:43:11.818402 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.816956 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae04339e-7f59-4edd-862a-aa88108d59bb-credential-socket\") pod \"e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w\" (UID: \"ae04339e-7f59-4edd-862a-aa88108d59bb\") " pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" Nov 02 23:43:11.818402 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.817032 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlj2x\" (UniqueName: \"kubernetes.io/projected/ae04339e-7f59-4edd-862a-aa88108d59bb-kube-api-access-qlj2x\") pod \"e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w\" (UID: \"ae04339e-7f59-4edd-862a-aa88108d59bb\") " pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" Nov 02 23:43:11.818402 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.817077 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae04339e-7f59-4edd-862a-aa88108d59bb-istiod-ca-cert\") pod \"e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w\" (UID: \"ae04339e-7f59-4edd-862a-aa88108d59bb\") " pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" Nov 02 23:43:11.818402 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.817129 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae04339e-7f59-4edd-862a-aa88108d59bb-istio-envoy\") pod \"e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w\" (UID: \"ae04339e-7f59-4edd-862a-aa88108d59bb\") " pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" Nov 02 23:43:11.818402 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.817171 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae04339e-7f59-4edd-862a-aa88108d59bb-istio-data\") pod \"e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w\" (UID: \"ae04339e-7f59-4edd-862a-aa88108d59bb\") " pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" Nov 02 23:43:11.818402 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.817218 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae04339e-7f59-4edd-862a-aa88108d59bb-istio-podinfo\") pod \"e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w\" (UID: \"ae04339e-7f59-4edd-862a-aa88108d59bb\") " pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" Nov 02 23:43:11.818402 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.817265 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae04339e-7f59-4edd-862a-aa88108d59bb-istio-token\") pod \"e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w\" (UID: \"ae04339e-7f59-4edd-862a-aa88108d59bb\") " pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" Nov 02 23:43:11.818402 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.817309 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae04339e-7f59-4edd-862a-aa88108d59bb-workload-socket\") pod \"e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w\" (UID: \"ae04339e-7f59-4edd-862a-aa88108d59bb\") " pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" Nov 02 23:43:11.818402 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.817373 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae04339e-7f59-4edd-862a-aa88108d59bb-workload-certs\") pod \"e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w\" (UID: \"ae04339e-7f59-4edd-862a-aa88108d59bb\") " pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" Nov 02 23:43:11.818402 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.817984 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae04339e-7f59-4edd-862a-aa88108d59bb-workload-certs\") pod \"e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w\" (UID: \"ae04339e-7f59-4edd-862a-aa88108d59bb\") " pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" Nov 02 23:43:11.819114 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.818433 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae04339e-7f59-4edd-862a-aa88108d59bb-credential-socket\") pod \"e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w\" (UID: \"ae04339e-7f59-4edd-862a-aa88108d59bb\") " pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" Nov 02 23:43:11.832982 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.819430 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae04339e-7f59-4edd-862a-aa88108d59bb-istiod-ca-cert\") pod \"e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w\" (UID: \"ae04339e-7f59-4edd-862a-aa88108d59bb\") " pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" Nov 02 23:43:11.832982 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.822210 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae04339e-7f59-4edd-862a-aa88108d59bb-istio-data\") pod \"e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w\" (UID: \"ae04339e-7f59-4edd-862a-aa88108d59bb\") " pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" Nov 02 23:43:11.859410 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.859372 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae04339e-7f59-4edd-862a-aa88108d59bb-workload-socket\") pod \"e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w\" (UID: \"ae04339e-7f59-4edd-862a-aa88108d59bb\") " pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" Nov 02 23:43:11.884917 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.881420 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlj2x\" (UniqueName: \"kubernetes.io/projected/ae04339e-7f59-4edd-862a-aa88108d59bb-kube-api-access-qlj2x\") pod \"e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w\" (UID: \"ae04339e-7f59-4edd-862a-aa88108d59bb\") " pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" Nov 02 23:43:11.896956 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.889206 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae04339e-7f59-4edd-862a-aa88108d59bb-istio-token\") pod \"e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w\" (UID: \"ae04339e-7f59-4edd-862a-aa88108d59bb\") " pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" Nov 02 23:43:11.911115 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.911032 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae04339e-7f59-4edd-862a-aa88108d59bb-istio-envoy\") pod \"e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w\" (UID: \"ae04339e-7f59-4edd-862a-aa88108d59bb\") " pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" Nov 02 23:43:11.912671 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.912641 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae04339e-7f59-4edd-862a-aa88108d59bb-istio-podinfo\") pod \"e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w\" (UID: \"ae04339e-7f59-4edd-862a-aa88108d59bb\") " pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" Nov 02 23:43:11.977329 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:11.977295 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" Nov 02 23:43:12.401230 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:12.401206 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w"] Nov 02 23:43:12.401731 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:43:12.401705 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae04339e_7f59_4edd_862a_aa88108d59bb.slice/crio-f741551e1646c106f2237badd39369ab106fa1d595c25b62a8857a29a6756adb WatchSource:0}: Error finding container f741551e1646c106f2237badd39369ab106fa1d595c25b62a8857a29a6756adb: Status 404 returned error can't find the container with id f741551e1646c106f2237badd39369ab106fa1d595c25b62a8857a29a6756adb Nov 02 23:43:12.404366 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:12.404331 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:43:12.404447 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:12.404396 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:43:12.404447 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:12.404425 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:43:13.186257 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:13.186220 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" event={"ID":"ae04339e-7f59-4edd-862a-aa88108d59bb","Type":"ContainerStarted","Data":"c3aeb1cc41a839182b6d2a514319eddfd8d4c5d26c0ce431bf6e568d14dd1e9f"} Nov 02 23:43:13.186752 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:13.186278 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" event={"ID":"ae04339e-7f59-4edd-862a-aa88108d59bb","Type":"ContainerStarted","Data":"f741551e1646c106f2237badd39369ab106fa1d595c25b62a8857a29a6756adb"} Nov 02 23:43:13.393834 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:13.393782 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" podStartSLOduration=2.393766817 podStartE2EDuration="2.393766817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:43:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:43:13.34632917 +0000 UTC m=+3042.314860953" watchObservedRunningTime="2026-11-02 23:43:13.393766817 +0000 UTC m=+3042.362298620" Nov 02 23:43:13.978342 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:13.978308 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" Nov 02 23:43:13.984441 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:13.984410 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" Nov 02 23:43:14.192900 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:14.192851 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" Nov 02 23:43:14.193945 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:14.193920 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" Nov 02 23:43:23.894117 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:23.894086 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc"] Nov 02 23:43:23.894590 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:23.894279 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" podUID="18e99da1-94f3-4d3e-a6ce-ee5a43219ce6" containerName="istio-proxy" containerID="cri-o://cb034058ecf6a2612b791cfc08edb7a3763969aa4a7f2611abcff23b2c6abf4c" gracePeriod=30 Nov 02 23:43:29.161463 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:29.161403 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" Nov 02 23:43:29.235175 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:29.235134 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-istio-envoy\") pod \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\" (UID: \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\") " Nov 02 23:43:29.235175 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:29.235174 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-istio-token\" (UniqueName: \"kubernetes.io/projected/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-istio-token\") pod \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\" (UID: \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\") " Nov 02 23:43:29.235382 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:29.235253 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-workload-socket\") pod \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\" (UID: \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\") " Nov 02 23:43:29.235382 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:29.235311 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-workload-certs\") pod \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\" (UID: \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\") " Nov 02 23:43:29.235382 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:29.235362 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-istiod-ca-cert\") pod \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\" (UID: \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\") " Nov 02 23:43:29.235526 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:29.235415 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-istio-data\") pod \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\" (UID: \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\") " Nov 02 23:43:29.235526 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:29.235483 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-kube-api-access-5v2bg\" (UniqueName: \"kubernetes.io/projected/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-kube-api-access-5v2bg\") pod \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\" (UID: \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\") " Nov 02 23:43:29.235646 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:29.235612 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-istio-podinfo\") pod \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\" (UID: \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\") " Nov 02 23:43:29.235714 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:29.235650 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-credential-socket\") pod \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\" (UID: \"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6\") " Nov 02 23:43:29.235714 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:29.235649 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-workload-certs" pod "18e99da1-94f3-4d3e-a6ce-ee5a43219ce6" (UID: "18e99da1-94f3-4d3e-a6ce-ee5a43219ce6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:29.235714 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:29.235662 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-workload-socket" pod "18e99da1-94f3-4d3e-a6ce-ee5a43219ce6" (UID: "18e99da1-94f3-4d3e-a6ce-ee5a43219ce6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:29.235871 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:29.235798 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-istiod-ca-cert" pod "18e99da1-94f3-4d3e-a6ce-ee5a43219ce6" (UID: "18e99da1-94f3-4d3e-a6ce-ee5a43219ce6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:43:29.235954 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:29.235914 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-istio-data" pod "18e99da1-94f3-4d3e-a6ce-ee5a43219ce6" (UID: "18e99da1-94f3-4d3e-a6ce-ee5a43219ce6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:29.236067 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:29.236033 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-credential-socket" pod "18e99da1-94f3-4d3e-a6ce-ee5a43219ce6" (UID: "18e99da1-94f3-4d3e-a6ce-ee5a43219ce6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:29.236451 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:29.236427 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:29.236451 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:29.236452 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:29.236644 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:29.236467 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:29.236644 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:29.236481 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:29.236644 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:29.236495 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:29.237613 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:29.237588 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-istio-envoy" pod "18e99da1-94f3-4d3e-a6ce-ee5a43219ce6" (UID: "18e99da1-94f3-4d3e-a6ce-ee5a43219ce6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:29.237713 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:29.237633 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-istio-token" pod "18e99da1-94f3-4d3e-a6ce-ee5a43219ce6" (UID: "18e99da1-94f3-4d3e-a6ce-ee5a43219ce6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:43:29.237713 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:29.237669 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-kube-api-access-5v2bg" pod "18e99da1-94f3-4d3e-a6ce-ee5a43219ce6" (UID: "18e99da1-94f3-4d3e-a6ce-ee5a43219ce6"). InnerVolumeSpecName "kube-api-access-5v2bg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:43:29.237967 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:29.237950 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-istio-podinfo" pod "18e99da1-94f3-4d3e-a6ce-ee5a43219ce6" (UID: "18e99da1-94f3-4d3e-a6ce-ee5a43219ce6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:43:29.287612 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:29.287581 5076 generic.go:356] "Generic (PLEG): container finished" podID="18e99da1-94f3-4d3e-a6ce-ee5a43219ce6" containerID="cb034058ecf6a2612b791cfc08edb7a3763969aa4a7f2611abcff23b2c6abf4c" exitCode=0 Nov 02 23:43:29.287762 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:29.287629 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" Nov 02 23:43:29.287762 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:29.287669 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" event={"ID":"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6","Type":"ContainerDied","Data":"cb034058ecf6a2612b791cfc08edb7a3763969aa4a7f2611abcff23b2c6abf4c"} Nov 02 23:43:29.287762 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:29.287721 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc" event={"ID":"18e99da1-94f3-4d3e-a6ce-ee5a43219ce6","Type":"ContainerDied","Data":"d3f1ec32bfb500659cc7c02d933d9ca0a0473715ad2a399cd3c86ad76a698af9"} Nov 02 23:43:29.287762 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:29.287742 5076 scope.go:122] "RemoveContainer" containerID="cb034058ecf6a2612b791cfc08edb7a3763969aa4a7f2611abcff23b2c6abf4c" Nov 02 23:43:29.298903 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:29.298872 5076 scope.go:122] "RemoveContainer" containerID="cb034058ecf6a2612b791cfc08edb7a3763969aa4a7f2611abcff23b2c6abf4c" Nov 02 23:43:29.299251 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:43:29.299228 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb034058ecf6a2612b791cfc08edb7a3763969aa4a7f2611abcff23b2c6abf4c\": container with ID starting with cb034058ecf6a2612b791cfc08edb7a3763969aa4a7f2611abcff23b2c6abf4c not found: ID does not exist" containerID="cb034058ecf6a2612b791cfc08edb7a3763969aa4a7f2611abcff23b2c6abf4c" Nov 02 23:43:29.299362 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:29.299258 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb034058ecf6a2612b791cfc08edb7a3763969aa4a7f2611abcff23b2c6abf4c"} err="failed to get container status \"cb034058ecf6a2612b791cfc08edb7a3763969aa4a7f2611abcff23b2c6abf4c\": rpc error: code = NotFound desc = could not find container \"cb034058ecf6a2612b791cfc08edb7a3763969aa4a7f2611abcff23b2c6abf4c\": container with ID starting with cb034058ecf6a2612b791cfc08edb7a3763969aa4a7f2611abcff23b2c6abf4c not found: ID does not exist" Nov 02 23:43:29.337702 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:29.337668 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v2bg\" (UniqueName: \"kubernetes.io/projected/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-kube-api-access-5v2bg\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:29.337702 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:29.337703 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:29.337966 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:29.337718 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:29.337966 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:29.337732 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:29.431675 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:29.431550 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc"] Nov 02 23:43:29.434712 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:29.434690 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-dw5dc"] Nov 02 23:43:31.156391 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:31.156354 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18e99da1-94f3-4d3e-a6ce-ee5a43219ce6" path="/var/lib/kubelet/pods/18e99da1-94f3-4d3e-a6ce-ee5a43219ce6/volumes" Nov 02 23:43:36.315474 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:36.315437 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-b4d53e/auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw"] Nov 02 23:43:36.316065 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:36.316048 5076 cpu_manager.go:414] "containerMap: removing container" podUID="18e99da1-94f3-4d3e-a6ce-ee5a43219ce6" containerName="istio-proxy" Nov 02 23:43:36.316116 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:36.316066 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18e99da1-94f3-4d3e-a6ce-ee5a43219ce6" containerName="istio-proxy" Nov 02 23:43:36.316116 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:36.316085 5076 cpu_manager.go:414] "containerMap: removing container" podUID="974b990b-722c-427a-9328-6f78b23b63d3" containerName="istio-proxy" Nov 02 23:43:36.316116 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:36.316090 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="974b990b-722c-427a-9328-6f78b23b63d3" containerName="istio-proxy" Nov 02 23:43:36.316230 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:36.316169 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="18e99da1-94f3-4d3e-a6ce-ee5a43219ce6" containerName="istio-proxy" Nov 02 23:43:36.320843 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:36.320822 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-b4d53e/auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw" Nov 02 23:43:36.374543 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:36.374509 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-b4d53e/auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw"] Nov 02 23:43:36.376322 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:36.376299 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-b4d53e\"/\"kube-root-ca.crt\"" Nov 02 23:43:36.376322 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:36.376313 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-b4d53e\"/\"openshift-service-ca.crt\"" Nov 02 23:43:36.418656 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:36.418623 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-b4d53e\"/\"auth-test-model-b4d53e-kserve-self-signed-certs\"" Nov 02 23:43:36.418831 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:36.418706 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-b4d53e\"/\"default-dockercfg-tvtlq\"" Nov 02 23:43:36.428360 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:36.428326 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c816427-1b10-4355-b096-264ce632a2aa-model-cache\") pod \"auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw\" (UID: \"8c816427-1b10-4355-b096-264ce632a2aa\") " pod="ai-tenant-e2e-shared-a-w4-b4d53e/auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw" Nov 02 23:43:36.428512 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:36.428477 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c816427-1b10-4355-b096-264ce632a2aa-tmp-dir\") pod \"auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw\" (UID: \"8c816427-1b10-4355-b096-264ce632a2aa\") " pod="ai-tenant-e2e-shared-a-w4-b4d53e/auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw" Nov 02 23:43:36.428565 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:36.428542 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c816427-1b10-4355-b096-264ce632a2aa-home\") pod \"auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw\" (UID: \"8c816427-1b10-4355-b096-264ce632a2aa\") " pod="ai-tenant-e2e-shared-a-w4-b4d53e/auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw" Nov 02 23:43:36.428643 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:36.428627 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c816427-1b10-4355-b096-264ce632a2aa-tls-certs\") pod \"auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw\" (UID: \"8c816427-1b10-4355-b096-264ce632a2aa\") " pod="ai-tenant-e2e-shared-a-w4-b4d53e/auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw" Nov 02 23:43:36.428811 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:36.428776 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c816427-1b10-4355-b096-264ce632a2aa-dshm\") pod \"auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw\" (UID: \"8c816427-1b10-4355-b096-264ce632a2aa\") " pod="ai-tenant-e2e-shared-a-w4-b4d53e/auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw" Nov 02 23:43:36.428995 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:36.428862 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzdzp\" (UniqueName: \"kubernetes.io/projected/8c816427-1b10-4355-b096-264ce632a2aa-kube-api-access-bzdzp\") pod \"auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw\" (UID: \"8c816427-1b10-4355-b096-264ce632a2aa\") " pod="ai-tenant-e2e-shared-a-w4-b4d53e/auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw" Nov 02 23:43:36.529770 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:36.529730 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c816427-1b10-4355-b096-264ce632a2aa-home\") pod \"auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw\" (UID: \"8c816427-1b10-4355-b096-264ce632a2aa\") " pod="ai-tenant-e2e-shared-a-w4-b4d53e/auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw" Nov 02 23:43:36.530007 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:36.529829 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c816427-1b10-4355-b096-264ce632a2aa-tls-certs\") pod \"auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw\" (UID: \"8c816427-1b10-4355-b096-264ce632a2aa\") " pod="ai-tenant-e2e-shared-a-w4-b4d53e/auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw" Nov 02 23:43:36.530007 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:36.529874 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c816427-1b10-4355-b096-264ce632a2aa-dshm\") pod \"auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw\" (UID: \"8c816427-1b10-4355-b096-264ce632a2aa\") " pod="ai-tenant-e2e-shared-a-w4-b4d53e/auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw" Nov 02 23:43:36.530007 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:36.529951 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzdzp\" (UniqueName: \"kubernetes.io/projected/8c816427-1b10-4355-b096-264ce632a2aa-kube-api-access-bzdzp\") pod \"auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw\" (UID: \"8c816427-1b10-4355-b096-264ce632a2aa\") " pod="ai-tenant-e2e-shared-a-w4-b4d53e/auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw" Nov 02 23:43:36.530156 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:36.530029 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c816427-1b10-4355-b096-264ce632a2aa-model-cache\") pod \"auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw\" (UID: \"8c816427-1b10-4355-b096-264ce632a2aa\") " pod="ai-tenant-e2e-shared-a-w4-b4d53e/auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw" Nov 02 23:43:36.530156 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:36.530139 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c816427-1b10-4355-b096-264ce632a2aa-tmp-dir\") pod \"auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw\" (UID: \"8c816427-1b10-4355-b096-264ce632a2aa\") " pod="ai-tenant-e2e-shared-a-w4-b4d53e/auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw" Nov 02 23:43:36.530236 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:36.530221 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c816427-1b10-4355-b096-264ce632a2aa-home\") pod \"auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw\" (UID: \"8c816427-1b10-4355-b096-264ce632a2aa\") " pod="ai-tenant-e2e-shared-a-w4-b4d53e/auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw" Nov 02 23:43:36.530413 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:36.530393 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c816427-1b10-4355-b096-264ce632a2aa-model-cache\") pod \"auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw\" (UID: \"8c816427-1b10-4355-b096-264ce632a2aa\") " pod="ai-tenant-e2e-shared-a-w4-b4d53e/auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw" Nov 02 23:43:36.530500 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:36.530476 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c816427-1b10-4355-b096-264ce632a2aa-tmp-dir\") pod \"auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw\" (UID: \"8c816427-1b10-4355-b096-264ce632a2aa\") " pod="ai-tenant-e2e-shared-a-w4-b4d53e/auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw" Nov 02 23:43:36.532162 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:36.532136 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c816427-1b10-4355-b096-264ce632a2aa-dshm\") pod \"auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw\" (UID: \"8c816427-1b10-4355-b096-264ce632a2aa\") " pod="ai-tenant-e2e-shared-a-w4-b4d53e/auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw" Nov 02 23:43:36.532465 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:36.532447 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c816427-1b10-4355-b096-264ce632a2aa-tls-certs\") pod \"auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw\" (UID: \"8c816427-1b10-4355-b096-264ce632a2aa\") " pod="ai-tenant-e2e-shared-a-w4-b4d53e/auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw" Nov 02 23:43:36.584284 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:36.584218 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzdzp\" (UniqueName: \"kubernetes.io/projected/8c816427-1b10-4355-b096-264ce632a2aa-kube-api-access-bzdzp\") pod \"auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw\" (UID: \"8c816427-1b10-4355-b096-264ce632a2aa\") " pod="ai-tenant-e2e-shared-a-w4-b4d53e/auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw" Nov 02 23:43:36.639287 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:36.639256 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-b4d53e/auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw" Nov 02 23:43:37.007536 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:37.007498 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-b4d53e/auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw"] Nov 02 23:43:37.008625 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:43:37.008586 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c816427_1b10_4355_b096_264ce632a2aa.slice/crio-6897c0a54d0a5382108bfc0153e4d8c24346a786c9e903f5fcf49db2c27ef76e WatchSource:0}: Error finding container 6897c0a54d0a5382108bfc0153e4d8c24346a786c9e903f5fcf49db2c27ef76e: Status 404 returned error can't find the container with id 6897c0a54d0a5382108bfc0153e4d8c24346a786c9e903f5fcf49db2c27ef76e Nov 02 23:43:37.344560 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:37.344467 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-b4d53e/auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw" event={"ID":"8c816427-1b10-4355-b096-264ce632a2aa","Type":"ContainerStarted","Data":"6f8dc9a364ae06e9fe04218d7b2d330dd5105edd5114d4e32bf76f5c14c07baf"} Nov 02 23:43:37.344560 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:37.344516 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-b4d53e/auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw" event={"ID":"8c816427-1b10-4355-b096-264ce632a2aa","Type":"ContainerStarted","Data":"6897c0a54d0a5382108bfc0153e4d8c24346a786c9e903f5fcf49db2c27ef76e"} Nov 02 23:43:37.454820 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:37.454765 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-b4d53e/auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw" podStartSLOduration=1.4547510670000001 podStartE2EDuration="1.454751067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:43:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:43:37.453350231 +0000 UTC m=+3066.421882016" watchObservedRunningTime="2026-11-02 23:43:37.454751067 +0000 UTC m=+3066.423282850" Nov 02 23:43:37.639684 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:37.639582 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-b4d53e/auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw" Nov 02 23:43:39.018374 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:39.018338 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5"] Nov 02 23:43:39.018883 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:39.018548 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" podUID="00d606e8-8e81-4b50-884b-d4e6bb36659c" containerName="istio-proxy" containerID="cri-o://813a3b75bfe8737b09a3cbba8a0c3f19069c4b5c4dc6c0f19aab2c73237b6011" gracePeriod=30 Nov 02 23:43:39.066437 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:39.066393 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9"] Nov 02 23:43:39.066678 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:39.066649 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" podUID="d61e6dcd-7996-4565-8481-aaafb4194707" containerName="istio-proxy" containerID="cri-o://e29511754dd30fa3434a74888a9ee05eacc79b4569c7da9629564a7e68c6e281" gracePeriod=30 Nov 02 23:43:39.069668 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:39.069634 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89"] Nov 02 23:43:39.069857 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:39.069823 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" podUID="c9975a71-d883-4d38-8b69-e3aa21574b13" containerName="istio-proxy" containerID="cri-o://4be028871e0b895ca7dcb8e38e7ca595eec460160446c8915ba0fb4b8b61ff74" gracePeriod=30 Nov 02 23:43:39.113510 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:39.113474 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5"] Nov 02 23:43:39.113800 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:39.113768 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" podUID="4db169a4-82e1-4ed5-b64c-adb40cab5629" containerName="istio-proxy" containerID="cri-o://f5165fb81e18cb08871e6a5af1ac049144a79bf53edbf7895178c3bb0e52048a" gracePeriod=30 Nov 02 23:43:39.883929 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:43:39.883875 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:43:44.068933 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.068904 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:43:44.069310 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.068955 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" podUID="c9975a71-d883-4d38-8b69-e3aa21574b13" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:43:44.225563 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.225520 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 02 23:43:44.225689 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.225603 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" podUID="d61e6dcd-7996-4565-8481-aaafb4194707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 02 23:43:44.353503 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.353473 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" Nov 02 23:43:44.425499 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.425466 5076 generic.go:356] "Generic (PLEG): container finished" podID="00d606e8-8e81-4b50-884b-d4e6bb36659c" containerID="813a3b75bfe8737b09a3cbba8a0c3f19069c4b5c4dc6c0f19aab2c73237b6011" exitCode=0 Nov 02 23:43:44.425667 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.425652 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" Nov 02 23:43:44.425809 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.425791 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" event={"ID":"00d606e8-8e81-4b50-884b-d4e6bb36659c","Type":"ContainerDied","Data":"813a3b75bfe8737b09a3cbba8a0c3f19069c4b5c4dc6c0f19aab2c73237b6011"} Nov 02 23:43:44.425885 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.425837 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" event={"ID":"00d606e8-8e81-4b50-884b-d4e6bb36659c","Type":"ContainerDied","Data":"124c0964ad0848e70df348f32f6a2a2e48250565944692525dbfd362e8dfe6a9"} Nov 02 23:43:44.425885 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.425871 5076 scope.go:122] "RemoveContainer" containerID="813a3b75bfe8737b09a3cbba8a0c3f19069c4b5c4dc6c0f19aab2c73237b6011" Nov 02 23:43:44.428697 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.428668 5076 generic.go:356] "Generic (PLEG): container finished" podID="c9975a71-d883-4d38-8b69-e3aa21574b13" containerID="4be028871e0b895ca7dcb8e38e7ca595eec460160446c8915ba0fb4b8b61ff74" exitCode=0 Nov 02 23:43:44.428833 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.428792 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" event={"ID":"c9975a71-d883-4d38-8b69-e3aa21574b13","Type":"ContainerDied","Data":"4be028871e0b895ca7dcb8e38e7ca595eec460160446c8915ba0fb4b8b61ff74"} Nov 02 23:43:44.431007 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.430982 5076 generic.go:356] "Generic (PLEG): container finished" podID="4db169a4-82e1-4ed5-b64c-adb40cab5629" containerID="f5165fb81e18cb08871e6a5af1ac049144a79bf53edbf7895178c3bb0e52048a" exitCode=0 Nov 02 23:43:44.431124 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.431060 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" event={"ID":"4db169a4-82e1-4ed5-b64c-adb40cab5629","Type":"ContainerDied","Data":"f5165fb81e18cb08871e6a5af1ac049144a79bf53edbf7895178c3bb0e52048a"} Nov 02 23:43:44.432827 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.432806 5076 generic.go:356] "Generic (PLEG): container finished" podID="d61e6dcd-7996-4565-8481-aaafb4194707" containerID="e29511754dd30fa3434a74888a9ee05eacc79b4569c7da9629564a7e68c6e281" exitCode=0 Nov 02 23:43:44.432969 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.432836 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" event={"ID":"d61e6dcd-7996-4565-8481-aaafb4194707","Type":"ContainerDied","Data":"e29511754dd30fa3434a74888a9ee05eacc79b4569c7da9629564a7e68c6e281"} Nov 02 23:43:44.434031 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.434004 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" Nov 02 23:43:44.439257 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.439231 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00d606e8-8e81-4b50-884b-d4e6bb36659c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00d606e8-8e81-4b50-884b-d4e6bb36659c-workload-certs\") pod \"00d606e8-8e81-4b50-884b-d4e6bb36659c\" (UID: \"00d606e8-8e81-4b50-884b-d4e6bb36659c\") " Nov 02 23:43:44.439380 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.439310 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00d606e8-8e81-4b50-884b-d4e6bb36659c-kube-api-access-29r8f\" (UniqueName: \"kubernetes.io/projected/00d606e8-8e81-4b50-884b-d4e6bb36659c-kube-api-access-29r8f\") pod \"00d606e8-8e81-4b50-884b-d4e6bb36659c\" (UID: \"00d606e8-8e81-4b50-884b-d4e6bb36659c\") " Nov 02 23:43:44.439380 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.439361 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00d606e8-8e81-4b50-884b-d4e6bb36659c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00d606e8-8e81-4b50-884b-d4e6bb36659c-istio-data\") pod \"00d606e8-8e81-4b50-884b-d4e6bb36659c\" (UID: \"00d606e8-8e81-4b50-884b-d4e6bb36659c\") " Nov 02 23:43:44.439516 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.439404 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00d606e8-8e81-4b50-884b-d4e6bb36659c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00d606e8-8e81-4b50-884b-d4e6bb36659c-istio-podinfo\") pod \"00d606e8-8e81-4b50-884b-d4e6bb36659c\" (UID: \"00d606e8-8e81-4b50-884b-d4e6bb36659c\") " Nov 02 23:43:44.439588 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.439530 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00d606e8-8e81-4b50-884b-d4e6bb36659c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d606e8-8e81-4b50-884b-d4e6bb36659c-workload-socket\") pod \"00d606e8-8e81-4b50-884b-d4e6bb36659c\" (UID: \"00d606e8-8e81-4b50-884b-d4e6bb36659c\") " Nov 02 23:43:44.439588 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.439572 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00d606e8-8e81-4b50-884b-d4e6bb36659c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00d606e8-8e81-4b50-884b-d4e6bb36659c-istio-envoy\") pod \"00d606e8-8e81-4b50-884b-d4e6bb36659c\" (UID: \"00d606e8-8e81-4b50-884b-d4e6bb36659c\") " Nov 02 23:43:44.439709 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.439613 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00d606e8-8e81-4b50-884b-d4e6bb36659c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00d606e8-8e81-4b50-884b-d4e6bb36659c-istiod-ca-cert\") pod \"00d606e8-8e81-4b50-884b-d4e6bb36659c\" (UID: \"00d606e8-8e81-4b50-884b-d4e6bb36659c\") " Nov 02 23:43:44.439709 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.439661 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00d606e8-8e81-4b50-884b-d4e6bb36659c-istio-token\" (UniqueName: \"kubernetes.io/projected/00d606e8-8e81-4b50-884b-d4e6bb36659c-istio-token\") pod \"00d606e8-8e81-4b50-884b-d4e6bb36659c\" (UID: \"00d606e8-8e81-4b50-884b-d4e6bb36659c\") " Nov 02 23:43:44.439709 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.439701 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00d606e8-8e81-4b50-884b-d4e6bb36659c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d606e8-8e81-4b50-884b-d4e6bb36659c-credential-socket\") pod \"00d606e8-8e81-4b50-884b-d4e6bb36659c\" (UID: \"00d606e8-8e81-4b50-884b-d4e6bb36659c\") " Nov 02 23:43:44.440530 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.440508 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d606e8-8e81-4b50-884b-d4e6bb36659c-credential-socket" pod "00d606e8-8e81-4b50-884b-d4e6bb36659c" (UID: "00d606e8-8e81-4b50-884b-d4e6bb36659c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:44.440715 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.440696 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d606e8-8e81-4b50-884b-d4e6bb36659c-workload-socket" pod "00d606e8-8e81-4b50-884b-d4e6bb36659c" (UID: "00d606e8-8e81-4b50-884b-d4e6bb36659c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:44.441268 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.441235 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d606e8-8e81-4b50-884b-d4e6bb36659c-istio-data" pod "00d606e8-8e81-4b50-884b-d4e6bb36659c" (UID: "00d606e8-8e81-4b50-884b-d4e6bb36659c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:44.441399 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.441365 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d606e8-8e81-4b50-884b-d4e6bb36659c-workload-certs" pod "00d606e8-8e81-4b50-884b-d4e6bb36659c" (UID: "00d606e8-8e81-4b50-884b-d4e6bb36659c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:44.441656 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.441633 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00d606e8-8e81-4b50-884b-d4e6bb36659c-istiod-ca-cert" pod "00d606e8-8e81-4b50-884b-d4e6bb36659c" (UID: "00d606e8-8e81-4b50-884b-d4e6bb36659c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:43:44.441841 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.441804 5076 scope.go:122] "RemoveContainer" containerID="813a3b75bfe8737b09a3cbba8a0c3f19069c4b5c4dc6c0f19aab2c73237b6011" Nov 02 23:43:44.442262 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:43:44.442163 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"813a3b75bfe8737b09a3cbba8a0c3f19069c4b5c4dc6c0f19aab2c73237b6011\": container with ID starting with 813a3b75bfe8737b09a3cbba8a0c3f19069c4b5c4dc6c0f19aab2c73237b6011 not found: ID does not exist" containerID="813a3b75bfe8737b09a3cbba8a0c3f19069c4b5c4dc6c0f19aab2c73237b6011" Nov 02 23:43:44.442262 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.442197 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"813a3b75bfe8737b09a3cbba8a0c3f19069c4b5c4dc6c0f19aab2c73237b6011"} err="failed to get container status \"813a3b75bfe8737b09a3cbba8a0c3f19069c4b5c4dc6c0f19aab2c73237b6011\": rpc error: code = NotFound desc = could not find container \"813a3b75bfe8737b09a3cbba8a0c3f19069c4b5c4dc6c0f19aab2c73237b6011\": container with ID starting with 813a3b75bfe8737b09a3cbba8a0c3f19069c4b5c4dc6c0f19aab2c73237b6011 not found: ID does not exist" Nov 02 23:43:44.444761 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.444727 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00d606e8-8e81-4b50-884b-d4e6bb36659c-istio-podinfo" pod "00d606e8-8e81-4b50-884b-d4e6bb36659c" (UID: "00d606e8-8e81-4b50-884b-d4e6bb36659c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:43:44.445834 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.445789 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00d606e8-8e81-4b50-884b-d4e6bb36659c-istio-token" pod "00d606e8-8e81-4b50-884b-d4e6bb36659c" (UID: "00d606e8-8e81-4b50-884b-d4e6bb36659c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:43:44.445963 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.445944 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00d606e8-8e81-4b50-884b-d4e6bb36659c-kube-api-access-29r8f" pod "00d606e8-8e81-4b50-884b-d4e6bb36659c" (UID: "00d606e8-8e81-4b50-884b-d4e6bb36659c"). InnerVolumeSpecName "kube-api-access-29r8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:43:44.446301 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.446277 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d606e8-8e81-4b50-884b-d4e6bb36659c-istio-envoy" pod "00d606e8-8e81-4b50-884b-d4e6bb36659c" (UID: "00d606e8-8e81-4b50-884b-d4e6bb36659c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:44.485768 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.485741 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" Nov 02 23:43:44.493214 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.493194 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" Nov 02 23:43:44.540938 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.540900 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d61e6dcd-7996-4565-8481-aaafb4194707-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61e6dcd-7996-4565-8481-aaafb4194707-istiod-ca-cert\") pod \"d61e6dcd-7996-4565-8481-aaafb4194707\" (UID: \"d61e6dcd-7996-4565-8481-aaafb4194707\") " Nov 02 23:43:44.541106 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.540950 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d61e6dcd-7996-4565-8481-aaafb4194707-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61e6dcd-7996-4565-8481-aaafb4194707-workload-certs\") pod \"d61e6dcd-7996-4565-8481-aaafb4194707\" (UID: \"d61e6dcd-7996-4565-8481-aaafb4194707\") " Nov 02 23:43:44.541106 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.540975 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d61e6dcd-7996-4565-8481-aaafb4194707-istio-token\" (UniqueName: \"kubernetes.io/projected/d61e6dcd-7996-4565-8481-aaafb4194707-istio-token\") pod \"d61e6dcd-7996-4565-8481-aaafb4194707\" (UID: \"d61e6dcd-7996-4565-8481-aaafb4194707\") " Nov 02 23:43:44.541106 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.541039 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d61e6dcd-7996-4565-8481-aaafb4194707-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61e6dcd-7996-4565-8481-aaafb4194707-istio-podinfo\") pod \"d61e6dcd-7996-4565-8481-aaafb4194707\" (UID: \"d61e6dcd-7996-4565-8481-aaafb4194707\") " Nov 02 23:43:44.541259 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.541144 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d61e6dcd-7996-4565-8481-aaafb4194707-kube-api-access-4wkpc\" (UniqueName: \"kubernetes.io/projected/d61e6dcd-7996-4565-8481-aaafb4194707-kube-api-access-4wkpc\") pod \"d61e6dcd-7996-4565-8481-aaafb4194707\" (UID: \"d61e6dcd-7996-4565-8481-aaafb4194707\") " Nov 02 23:43:44.541259 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.541234 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d61e6dcd-7996-4565-8481-aaafb4194707-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61e6dcd-7996-4565-8481-aaafb4194707-workload-socket\") pod \"d61e6dcd-7996-4565-8481-aaafb4194707\" (UID: \"d61e6dcd-7996-4565-8481-aaafb4194707\") " Nov 02 23:43:44.541376 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.541259 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d61e6dcd-7996-4565-8481-aaafb4194707-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61e6dcd-7996-4565-8481-aaafb4194707-istio-envoy\") pod \"d61e6dcd-7996-4565-8481-aaafb4194707\" (UID: \"d61e6dcd-7996-4565-8481-aaafb4194707\") " Nov 02 23:43:44.541376 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.541265 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d61e6dcd-7996-4565-8481-aaafb4194707-istiod-ca-cert" pod "d61e6dcd-7996-4565-8481-aaafb4194707" (UID: "d61e6dcd-7996-4565-8481-aaafb4194707"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:43:44.541376 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.541266 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61e6dcd-7996-4565-8481-aaafb4194707-workload-certs" pod "d61e6dcd-7996-4565-8481-aaafb4194707" (UID: "d61e6dcd-7996-4565-8481-aaafb4194707"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:44.541376 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.541318 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d61e6dcd-7996-4565-8481-aaafb4194707-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61e6dcd-7996-4565-8481-aaafb4194707-credential-socket\") pod \"d61e6dcd-7996-4565-8481-aaafb4194707\" (UID: \"d61e6dcd-7996-4565-8481-aaafb4194707\") " Nov 02 23:43:44.541610 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.541392 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d61e6dcd-7996-4565-8481-aaafb4194707-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61e6dcd-7996-4565-8481-aaafb4194707-istio-data\") pod \"d61e6dcd-7996-4565-8481-aaafb4194707\" (UID: \"d61e6dcd-7996-4565-8481-aaafb4194707\") " Nov 02 23:43:44.541610 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.541481 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61e6dcd-7996-4565-8481-aaafb4194707-workload-socket" pod "d61e6dcd-7996-4565-8481-aaafb4194707" (UID: "d61e6dcd-7996-4565-8481-aaafb4194707"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:44.541713 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.541622 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61e6dcd-7996-4565-8481-aaafb4194707-credential-socket" pod "d61e6dcd-7996-4565-8481-aaafb4194707" (UID: "d61e6dcd-7996-4565-8481-aaafb4194707"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:44.541828 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.541794 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61e6dcd-7996-4565-8481-aaafb4194707-istio-data" pod "d61e6dcd-7996-4565-8481-aaafb4194707" (UID: "d61e6dcd-7996-4565-8481-aaafb4194707"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:44.542012 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.541986 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29r8f\" (UniqueName: \"kubernetes.io/projected/00d606e8-8e81-4b50-884b-d4e6bb36659c-kube-api-access-29r8f\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.543345 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.542016 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00d606e8-8e81-4b50-884b-d4e6bb36659c-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.543345 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.542033 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00d606e8-8e81-4b50-884b-d4e6bb36659c-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.543345 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.542046 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61e6dcd-7996-4565-8481-aaafb4194707-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.543345 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.542060 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61e6dcd-7996-4565-8481-aaafb4194707-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.543345 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.542074 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d606e8-8e81-4b50-884b-d4e6bb36659c-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.543345 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.542088 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00d606e8-8e81-4b50-884b-d4e6bb36659c-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.543345 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.542102 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00d606e8-8e81-4b50-884b-d4e6bb36659c-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.543345 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.542116 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61e6dcd-7996-4565-8481-aaafb4194707-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.543345 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.542130 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00d606e8-8e81-4b50-884b-d4e6bb36659c-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.543345 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.542145 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d606e8-8e81-4b50-884b-d4e6bb36659c-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.543345 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.542159 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61e6dcd-7996-4565-8481-aaafb4194707-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.543345 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.542174 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61e6dcd-7996-4565-8481-aaafb4194707-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.543345 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.542187 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00d606e8-8e81-4b50-884b-d4e6bb36659c-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.544255 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.543612 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d61e6dcd-7996-4565-8481-aaafb4194707-kube-api-access-4wkpc" pod "d61e6dcd-7996-4565-8481-aaafb4194707" (UID: "d61e6dcd-7996-4565-8481-aaafb4194707"). InnerVolumeSpecName "kube-api-access-4wkpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:43:44.544255 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.543635 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d61e6dcd-7996-4565-8481-aaafb4194707-istio-podinfo" pod "d61e6dcd-7996-4565-8481-aaafb4194707" (UID: "d61e6dcd-7996-4565-8481-aaafb4194707"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:43:44.544255 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.543713 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d61e6dcd-7996-4565-8481-aaafb4194707-istio-token" pod "d61e6dcd-7996-4565-8481-aaafb4194707" (UID: "d61e6dcd-7996-4565-8481-aaafb4194707"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:43:44.544255 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.543835 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61e6dcd-7996-4565-8481-aaafb4194707-istio-envoy" pod "d61e6dcd-7996-4565-8481-aaafb4194707" (UID: "d61e6dcd-7996-4565-8481-aaafb4194707"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:44.642957 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.642840 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9975a71-d883-4d38-8b69-e3aa21574b13-kube-api-access-crmh9\" (UniqueName: \"kubernetes.io/projected/c9975a71-d883-4d38-8b69-e3aa21574b13-kube-api-access-crmh9\") pod \"c9975a71-d883-4d38-8b69-e3aa21574b13\" (UID: \"c9975a71-d883-4d38-8b69-e3aa21574b13\") " Nov 02 23:43:44.642957 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.642879 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9975a71-d883-4d38-8b69-e3aa21574b13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9975a71-d883-4d38-8b69-e3aa21574b13-workload-certs\") pod \"c9975a71-d883-4d38-8b69-e3aa21574b13\" (UID: \"c9975a71-d883-4d38-8b69-e3aa21574b13\") " Nov 02 23:43:44.642957 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.642921 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9975a71-d883-4d38-8b69-e3aa21574b13-istio-token\" (UniqueName: \"kubernetes.io/projected/c9975a71-d883-4d38-8b69-e3aa21574b13-istio-token\") pod \"c9975a71-d883-4d38-8b69-e3aa21574b13\" (UID: \"c9975a71-d883-4d38-8b69-e3aa21574b13\") " Nov 02 23:43:44.643245 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.642960 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4db169a4-82e1-4ed5-b64c-adb40cab5629-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4db169a4-82e1-4ed5-b64c-adb40cab5629-istio-data\") pod \"4db169a4-82e1-4ed5-b64c-adb40cab5629\" (UID: \"4db169a4-82e1-4ed5-b64c-adb40cab5629\") " Nov 02 23:43:44.643245 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.642993 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4db169a4-82e1-4ed5-b64c-adb40cab5629-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4db169a4-82e1-4ed5-b64c-adb40cab5629-istio-podinfo\") pod \"4db169a4-82e1-4ed5-b64c-adb40cab5629\" (UID: \"4db169a4-82e1-4ed5-b64c-adb40cab5629\") " Nov 02 23:43:44.643245 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.643023 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4db169a4-82e1-4ed5-b64c-adb40cab5629-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4db169a4-82e1-4ed5-b64c-adb40cab5629-istio-envoy\") pod \"4db169a4-82e1-4ed5-b64c-adb40cab5629\" (UID: \"4db169a4-82e1-4ed5-b64c-adb40cab5629\") " Nov 02 23:43:44.643245 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.643052 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9975a71-d883-4d38-8b69-e3aa21574b13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9975a71-d883-4d38-8b69-e3aa21574b13-istio-envoy\") pod \"c9975a71-d883-4d38-8b69-e3aa21574b13\" (UID: \"c9975a71-d883-4d38-8b69-e3aa21574b13\") " Nov 02 23:43:44.643245 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.643109 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4db169a4-82e1-4ed5-b64c-adb40cab5629-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4db169a4-82e1-4ed5-b64c-adb40cab5629-workload-certs\") pod \"4db169a4-82e1-4ed5-b64c-adb40cab5629\" (UID: \"4db169a4-82e1-4ed5-b64c-adb40cab5629\") " Nov 02 23:43:44.643245 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.643141 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4db169a4-82e1-4ed5-b64c-adb40cab5629-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db169a4-82e1-4ed5-b64c-adb40cab5629-workload-socket\") pod \"4db169a4-82e1-4ed5-b64c-adb40cab5629\" (UID: \"4db169a4-82e1-4ed5-b64c-adb40cab5629\") " Nov 02 23:43:44.643245 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.643166 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9975a71-d883-4d38-8b69-e3aa21574b13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9975a71-d883-4d38-8b69-e3aa21574b13-istiod-ca-cert\") pod \"c9975a71-d883-4d38-8b69-e3aa21574b13\" (UID: \"c9975a71-d883-4d38-8b69-e3aa21574b13\") " Nov 02 23:43:44.643245 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.643200 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9975a71-d883-4d38-8b69-e3aa21574b13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9975a71-d883-4d38-8b69-e3aa21574b13-credential-socket\") pod \"c9975a71-d883-4d38-8b69-e3aa21574b13\" (UID: \"c9975a71-d883-4d38-8b69-e3aa21574b13\") " Nov 02 23:43:44.643245 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.643233 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4db169a4-82e1-4ed5-b64c-adb40cab5629-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4db169a4-82e1-4ed5-b64c-adb40cab5629-istiod-ca-cert\") pod \"4db169a4-82e1-4ed5-b64c-adb40cab5629\" (UID: \"4db169a4-82e1-4ed5-b64c-adb40cab5629\") " Nov 02 23:43:44.643732 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.643297 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9975a71-d883-4d38-8b69-e3aa21574b13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9975a71-d883-4d38-8b69-e3aa21574b13-istio-data\") pod \"c9975a71-d883-4d38-8b69-e3aa21574b13\" (UID: \"c9975a71-d883-4d38-8b69-e3aa21574b13\") " Nov 02 23:43:44.643732 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.643333 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9975a71-d883-4d38-8b69-e3aa21574b13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9975a71-d883-4d38-8b69-e3aa21574b13-istio-podinfo\") pod \"c9975a71-d883-4d38-8b69-e3aa21574b13\" (UID: \"c9975a71-d883-4d38-8b69-e3aa21574b13\") " Nov 02 23:43:44.643732 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.643486 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9975a71-d883-4d38-8b69-e3aa21574b13-workload-certs" pod "c9975a71-d883-4d38-8b69-e3aa21574b13" (UID: "c9975a71-d883-4d38-8b69-e3aa21574b13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:44.643732 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.643490 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db169a4-82e1-4ed5-b64c-adb40cab5629-istio-data" pod "4db169a4-82e1-4ed5-b64c-adb40cab5629" (UID: "4db169a4-82e1-4ed5-b64c-adb40cab5629"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:44.643732 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.643519 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9975a71-d883-4d38-8b69-e3aa21574b13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9975a71-d883-4d38-8b69-e3aa21574b13-workload-socket\") pod \"c9975a71-d883-4d38-8b69-e3aa21574b13\" (UID: \"c9975a71-d883-4d38-8b69-e3aa21574b13\") " Nov 02 23:43:44.643732 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.643578 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4db169a4-82e1-4ed5-b64c-adb40cab5629-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db169a4-82e1-4ed5-b64c-adb40cab5629-credential-socket\") pod \"4db169a4-82e1-4ed5-b64c-adb40cab5629\" (UID: \"4db169a4-82e1-4ed5-b64c-adb40cab5629\") " Nov 02 23:43:44.643732 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.643627 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4db169a4-82e1-4ed5-b64c-adb40cab5629-kube-api-access-mdt2p\" (UniqueName: \"kubernetes.io/projected/4db169a4-82e1-4ed5-b64c-adb40cab5629-kube-api-access-mdt2p\") pod \"4db169a4-82e1-4ed5-b64c-adb40cab5629\" (UID: \"4db169a4-82e1-4ed5-b64c-adb40cab5629\") " Nov 02 23:43:44.643732 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.643698 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4db169a4-82e1-4ed5-b64c-adb40cab5629-istio-token\" (UniqueName: \"kubernetes.io/projected/4db169a4-82e1-4ed5-b64c-adb40cab5629-istio-token\") pod \"4db169a4-82e1-4ed5-b64c-adb40cab5629\" (UID: \"4db169a4-82e1-4ed5-b64c-adb40cab5629\") " Nov 02 23:43:44.644324 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.644303 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d61e6dcd-7996-4565-8481-aaafb4194707-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.644492 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.644477 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61e6dcd-7996-4565-8481-aaafb4194707-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.644597 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.644583 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9975a71-d883-4d38-8b69-e3aa21574b13-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.644697 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.644682 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wkpc\" (UniqueName: \"kubernetes.io/projected/d61e6dcd-7996-4565-8481-aaafb4194707-kube-api-access-4wkpc\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.644798 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.644786 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4db169a4-82e1-4ed5-b64c-adb40cab5629-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.644913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.644883 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61e6dcd-7996-4565-8481-aaafb4194707-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.645014 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.643676 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db169a4-82e1-4ed5-b64c-adb40cab5629-workload-socket" pod "4db169a4-82e1-4ed5-b64c-adb40cab5629" (UID: "4db169a4-82e1-4ed5-b64c-adb40cab5629"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:44.645114 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.643820 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4db169a4-82e1-4ed5-b64c-adb40cab5629-istiod-ca-cert" pod "4db169a4-82e1-4ed5-b64c-adb40cab5629" (UID: "4db169a4-82e1-4ed5-b64c-adb40cab5629"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:43:44.646715 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.646686 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db169a4-82e1-4ed5-b64c-adb40cab5629-credential-socket" pod "4db169a4-82e1-4ed5-b64c-adb40cab5629" (UID: "4db169a4-82e1-4ed5-b64c-adb40cab5629"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:44.647541 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.643827 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9975a71-d883-4d38-8b69-e3aa21574b13-istiod-ca-cert" pod "c9975a71-d883-4d38-8b69-e3aa21574b13" (UID: "c9975a71-d883-4d38-8b69-e3aa21574b13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:43:44.647671 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.644086 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9975a71-d883-4d38-8b69-e3aa21574b13-credential-socket" pod "c9975a71-d883-4d38-8b69-e3aa21574b13" (UID: "c9975a71-d883-4d38-8b69-e3aa21574b13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:44.647671 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.644238 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db169a4-82e1-4ed5-b64c-adb40cab5629-workload-certs" pod "4db169a4-82e1-4ed5-b64c-adb40cab5629" (UID: "4db169a4-82e1-4ed5-b64c-adb40cab5629"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:44.647671 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.644394 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9975a71-d883-4d38-8b69-e3aa21574b13-workload-socket" pod "c9975a71-d883-4d38-8b69-e3aa21574b13" (UID: "c9975a71-d883-4d38-8b69-e3aa21574b13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:44.647913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.644558 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9975a71-d883-4d38-8b69-e3aa21574b13-istio-data" pod "c9975a71-d883-4d38-8b69-e3aa21574b13" (UID: "c9975a71-d883-4d38-8b69-e3aa21574b13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:44.647913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.647687 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9975a71-d883-4d38-8b69-e3aa21574b13-istio-podinfo" pod "c9975a71-d883-4d38-8b69-e3aa21574b13" (UID: "c9975a71-d883-4d38-8b69-e3aa21574b13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:43:44.648932 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.648870 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9975a71-d883-4d38-8b69-e3aa21574b13-istio-envoy" pod "c9975a71-d883-4d38-8b69-e3aa21574b13" (UID: "c9975a71-d883-4d38-8b69-e3aa21574b13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:44.650062 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.650036 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4db169a4-82e1-4ed5-b64c-adb40cab5629-istio-token" pod "4db169a4-82e1-4ed5-b64c-adb40cab5629" (UID: "4db169a4-82e1-4ed5-b64c-adb40cab5629"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:43:44.650167 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.650086 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4db169a4-82e1-4ed5-b64c-adb40cab5629-kube-api-access-mdt2p" pod "4db169a4-82e1-4ed5-b64c-adb40cab5629" (UID: "4db169a4-82e1-4ed5-b64c-adb40cab5629"). InnerVolumeSpecName "kube-api-access-mdt2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:43:44.650167 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.650138 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db169a4-82e1-4ed5-b64c-adb40cab5629-istio-envoy" pod "4db169a4-82e1-4ed5-b64c-adb40cab5629" (UID: "4db169a4-82e1-4ed5-b64c-adb40cab5629"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:44.650293 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.650169 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9975a71-d883-4d38-8b69-e3aa21574b13-istio-token" pod "c9975a71-d883-4d38-8b69-e3aa21574b13" (UID: "c9975a71-d883-4d38-8b69-e3aa21574b13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:43:44.650293 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.650275 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4db169a4-82e1-4ed5-b64c-adb40cab5629-istio-podinfo" pod "4db169a4-82e1-4ed5-b64c-adb40cab5629" (UID: "4db169a4-82e1-4ed5-b64c-adb40cab5629"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:43:44.651686 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.651659 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9975a71-d883-4d38-8b69-e3aa21574b13-kube-api-access-crmh9" pod "c9975a71-d883-4d38-8b69-e3aa21574b13" (UID: "c9975a71-d883-4d38-8b69-e3aa21574b13"). InnerVolumeSpecName "kube-api-access-crmh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:43:44.751610 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.751572 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4db169a4-82e1-4ed5-b64c-adb40cab5629-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.751610 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.751610 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crmh9\" (UniqueName: \"kubernetes.io/projected/c9975a71-d883-4d38-8b69-e3aa21574b13-kube-api-access-crmh9\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.751864 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.751628 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9975a71-d883-4d38-8b69-e3aa21574b13-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.751864 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.751643 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4db169a4-82e1-4ed5-b64c-adb40cab5629-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.751864 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.751661 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4db169a4-82e1-4ed5-b64c-adb40cab5629-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.751864 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.751682 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9975a71-d883-4d38-8b69-e3aa21574b13-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.751864 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.751695 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4db169a4-82e1-4ed5-b64c-adb40cab5629-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.751864 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.751708 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db169a4-82e1-4ed5-b64c-adb40cab5629-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.751864 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.751726 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9975a71-d883-4d38-8b69-e3aa21574b13-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.751864 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.751739 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9975a71-d883-4d38-8b69-e3aa21574b13-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.751864 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.751752 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4db169a4-82e1-4ed5-b64c-adb40cab5629-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.751864 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.751766 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9975a71-d883-4d38-8b69-e3aa21574b13-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.751864 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.751783 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9975a71-d883-4d38-8b69-e3aa21574b13-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.751864 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.751795 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9975a71-d883-4d38-8b69-e3aa21574b13-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.751864 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.751808 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db169a4-82e1-4ed5-b64c-adb40cab5629-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.751864 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:44.751822 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdt2p\" (UniqueName: \"kubernetes.io/projected/4db169a4-82e1-4ed5-b64c-adb40cab5629-kube-api-access-mdt2p\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:45.020350 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:45.020313 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5"] Nov 02 23:43:45.023158 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:45.023130 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5"] Nov 02 23:43:45.152147 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:45.152114 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00d606e8-8e81-4b50-884b-d4e6bb36659c" path="/var/lib/kubelet/pods/00d606e8-8e81-4b50-884b-d4e6bb36659c/volumes" Nov 02 23:43:45.261343 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:45.261305 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:43:45.261494 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:45.261365 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-cknl5" podUID="00d606e8-8e81-4b50-884b-d4e6bb36659c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:43:45.440905 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:45.440863 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" Nov 02 23:43:45.440905 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:45.440880 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9" event={"ID":"d61e6dcd-7996-4565-8481-aaafb4194707","Type":"ContainerDied","Data":"5a8ef275495ff145d7e24678279b261ce5ebce9f7cd45e7836bc3cfca75ed80a"} Nov 02 23:43:45.441143 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:45.440972 5076 scope.go:122] "RemoveContainer" containerID="e29511754dd30fa3434a74888a9ee05eacc79b4569c7da9629564a7e68c6e281" Nov 02 23:43:45.443735 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:45.443716 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" Nov 02 23:43:45.443735 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:45.443722 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89" event={"ID":"c9975a71-d883-4d38-8b69-e3aa21574b13","Type":"ContainerDied","Data":"001944fb8e4a8891bf2c12ae3911c0e3f30de18eab85e0511c2b00c7888a6b44"} Nov 02 23:43:45.445300 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:45.445284 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" Nov 02 23:43:45.445418 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:45.445282 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5" event={"ID":"4db169a4-82e1-4ed5-b64c-adb40cab5629","Type":"ContainerDied","Data":"0c6c86b977337c36e057996f9d8b05af4e179ea1733185670455645a80c47dda"} Nov 02 23:43:45.451783 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:45.451760 5076 scope.go:122] "RemoveContainer" containerID="4be028871e0b895ca7dcb8e38e7ca595eec460160446c8915ba0fb4b8b61ff74" Nov 02 23:43:45.460578 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:45.460560 5076 scope.go:122] "RemoveContainer" containerID="f5165fb81e18cb08871e6a5af1ac049144a79bf53edbf7895178c3bb0e52048a" Nov 02 23:43:45.597755 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:45.597715 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89"] Nov 02 23:43:45.603249 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:45.603217 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-nms89"] Nov 02 23:43:45.708180 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:45.708098 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5"] Nov 02 23:43:45.710874 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:45.710851 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-n4jn5"] Nov 02 23:43:45.816371 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:45.816336 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9"] Nov 02 23:43:45.819224 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:45.819203 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-rs6x9"] Nov 02 23:43:46.640417 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:46.640387 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-b4d53e/auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw" Nov 02 23:43:46.652982 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:46.652957 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-b4d53e/auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw" Nov 02 23:43:47.148683 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:47.148648 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4db169a4-82e1-4ed5-b64c-adb40cab5629" path="/var/lib/kubelet/pods/4db169a4-82e1-4ed5-b64c-adb40cab5629/volumes" Nov 02 23:43:47.149200 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:47.149185 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9975a71-d883-4d38-8b69-e3aa21574b13" path="/var/lib/kubelet/pods/c9975a71-d883-4d38-8b69-e3aa21574b13/volumes" Nov 02 23:43:47.149595 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:47.149582 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d61e6dcd-7996-4565-8481-aaafb4194707" path="/var/lib/kubelet/pods/d61e6dcd-7996-4565-8481-aaafb4194707/volumes" Nov 02 23:43:47.472734 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:47.472706 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-b4d53e/auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw" Nov 02 23:43:49.831059 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:49.831023 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-fbec82/auth-test-model-fbec82-kserve-566f67c557-f9g2v"] Nov 02 23:43:49.831599 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:49.831583 5076 cpu_manager.go:414] "containerMap: removing container" podUID="d61e6dcd-7996-4565-8481-aaafb4194707" containerName="istio-proxy" Nov 02 23:43:49.831657 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:49.831600 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d61e6dcd-7996-4565-8481-aaafb4194707" containerName="istio-proxy" Nov 02 23:43:49.831657 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:49.831634 5076 cpu_manager.go:414] "containerMap: removing container" podUID="4db169a4-82e1-4ed5-b64c-adb40cab5629" containerName="istio-proxy" Nov 02 23:43:49.831657 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:49.831639 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4db169a4-82e1-4ed5-b64c-adb40cab5629" containerName="istio-proxy" Nov 02 23:43:49.831657 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:49.831652 5076 cpu_manager.go:414] "containerMap: removing container" podUID="c9975a71-d883-4d38-8b69-e3aa21574b13" containerName="istio-proxy" Nov 02 23:43:49.831657 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:49.831657 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9975a71-d883-4d38-8b69-e3aa21574b13" containerName="istio-proxy" Nov 02 23:43:49.831829 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:49.831671 5076 cpu_manager.go:414] "containerMap: removing container" podUID="00d606e8-8e81-4b50-884b-d4e6bb36659c" containerName="istio-proxy" Nov 02 23:43:49.831829 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:49.831676 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00d606e8-8e81-4b50-884b-d4e6bb36659c" containerName="istio-proxy" Nov 02 23:43:49.831829 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:49.831756 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9975a71-d883-4d38-8b69-e3aa21574b13" containerName="istio-proxy" Nov 02 23:43:49.831829 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:49.831767 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="00d606e8-8e81-4b50-884b-d4e6bb36659c" containerName="istio-proxy" Nov 02 23:43:49.831829 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:49.831774 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="d61e6dcd-7996-4565-8481-aaafb4194707" containerName="istio-proxy" Nov 02 23:43:49.831829 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:49.831785 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="4db169a4-82e1-4ed5-b64c-adb40cab5629" containerName="istio-proxy" Nov 02 23:43:49.840115 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:49.840095 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-fbec82/auth-test-model-fbec82-kserve-566f67c557-f9g2v" Nov 02 23:43:49.888515 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:49.888483 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-fbec82/auth-test-model-fbec82-kserve-566f67c557-f9g2v"] Nov 02 23:43:49.917784 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:49.917746 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzdm6\" (UniqueName: \"kubernetes.io/projected/7037c73c-59cf-4f54-a054-201b076b62cb-kube-api-access-kzdm6\") pod \"auth-test-model-fbec82-kserve-566f67c557-f9g2v\" (UID: \"7037c73c-59cf-4f54-a054-201b076b62cb\") " pod="ai-tenant-e2e-shared-b-w2-fbec82/auth-test-model-fbec82-kserve-566f67c557-f9g2v" Nov 02 23:43:49.917972 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:49.917813 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7037c73c-59cf-4f54-a054-201b076b62cb-tls-certs\") pod \"auth-test-model-fbec82-kserve-566f67c557-f9g2v\" (UID: \"7037c73c-59cf-4f54-a054-201b076b62cb\") " pod="ai-tenant-e2e-shared-b-w2-fbec82/auth-test-model-fbec82-kserve-566f67c557-f9g2v" Nov 02 23:43:49.917972 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:49.917953 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7037c73c-59cf-4f54-a054-201b076b62cb-home\") pod \"auth-test-model-fbec82-kserve-566f67c557-f9g2v\" (UID: \"7037c73c-59cf-4f54-a054-201b076b62cb\") " pod="ai-tenant-e2e-shared-b-w2-fbec82/auth-test-model-fbec82-kserve-566f67c557-f9g2v" Nov 02 23:43:49.918065 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:49.917982 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7037c73c-59cf-4f54-a054-201b076b62cb-tmp-dir\") pod \"auth-test-model-fbec82-kserve-566f67c557-f9g2v\" (UID: \"7037c73c-59cf-4f54-a054-201b076b62cb\") " pod="ai-tenant-e2e-shared-b-w2-fbec82/auth-test-model-fbec82-kserve-566f67c557-f9g2v" Nov 02 23:43:49.918112 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:49.918082 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7037c73c-59cf-4f54-a054-201b076b62cb-dshm\") pod \"auth-test-model-fbec82-kserve-566f67c557-f9g2v\" (UID: \"7037c73c-59cf-4f54-a054-201b076b62cb\") " pod="ai-tenant-e2e-shared-b-w2-fbec82/auth-test-model-fbec82-kserve-566f67c557-f9g2v" Nov 02 23:43:49.918151 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:49.918113 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7037c73c-59cf-4f54-a054-201b076b62cb-model-cache\") pod \"auth-test-model-fbec82-kserve-566f67c557-f9g2v\" (UID: \"7037c73c-59cf-4f54-a054-201b076b62cb\") " pod="ai-tenant-e2e-shared-b-w2-fbec82/auth-test-model-fbec82-kserve-566f67c557-f9g2v" Nov 02 23:43:49.932416 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:49.932383 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-fbec82\"/\"openshift-service-ca.crt\"" Nov 02 23:43:49.932546 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:49.932431 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-fbec82\"/\"kube-root-ca.crt\"" Nov 02 23:43:49.936867 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:49.936849 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-fbec82\"/\"auth-test-model-fbec82-kserve-self-signed-certs\"" Nov 02 23:43:49.936965 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:49.936851 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-fbec82\"/\"default-dockercfg-wrlmq\"" Nov 02 23:43:50.025818 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:50.021057 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7037c73c-59cf-4f54-a054-201b076b62cb-tls-certs\") pod \"auth-test-model-fbec82-kserve-566f67c557-f9g2v\" (UID: \"7037c73c-59cf-4f54-a054-201b076b62cb\") " pod="ai-tenant-e2e-shared-b-w2-fbec82/auth-test-model-fbec82-kserve-566f67c557-f9g2v" Nov 02 23:43:50.025818 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:50.021150 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7037c73c-59cf-4f54-a054-201b076b62cb-home\") pod \"auth-test-model-fbec82-kserve-566f67c557-f9g2v\" (UID: \"7037c73c-59cf-4f54-a054-201b076b62cb\") " pod="ai-tenant-e2e-shared-b-w2-fbec82/auth-test-model-fbec82-kserve-566f67c557-f9g2v" Nov 02 23:43:50.025818 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:50.021180 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7037c73c-59cf-4f54-a054-201b076b62cb-tmp-dir\") pod \"auth-test-model-fbec82-kserve-566f67c557-f9g2v\" (UID: \"7037c73c-59cf-4f54-a054-201b076b62cb\") " pod="ai-tenant-e2e-shared-b-w2-fbec82/auth-test-model-fbec82-kserve-566f67c557-f9g2v" Nov 02 23:43:50.025818 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:50.021253 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7037c73c-59cf-4f54-a054-201b076b62cb-dshm\") pod \"auth-test-model-fbec82-kserve-566f67c557-f9g2v\" (UID: \"7037c73c-59cf-4f54-a054-201b076b62cb\") " pod="ai-tenant-e2e-shared-b-w2-fbec82/auth-test-model-fbec82-kserve-566f67c557-f9g2v" Nov 02 23:43:50.025818 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:50.021299 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7037c73c-59cf-4f54-a054-201b076b62cb-model-cache\") pod \"auth-test-model-fbec82-kserve-566f67c557-f9g2v\" (UID: \"7037c73c-59cf-4f54-a054-201b076b62cb\") " pod="ai-tenant-e2e-shared-b-w2-fbec82/auth-test-model-fbec82-kserve-566f67c557-f9g2v" Nov 02 23:43:50.025818 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:50.021427 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzdm6\" (UniqueName: \"kubernetes.io/projected/7037c73c-59cf-4f54-a054-201b076b62cb-kube-api-access-kzdm6\") pod \"auth-test-model-fbec82-kserve-566f67c557-f9g2v\" (UID: \"7037c73c-59cf-4f54-a054-201b076b62cb\") " pod="ai-tenant-e2e-shared-b-w2-fbec82/auth-test-model-fbec82-kserve-566f67c557-f9g2v" Nov 02 23:43:50.025818 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:50.023567 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7037c73c-59cf-4f54-a054-201b076b62cb-home\") pod \"auth-test-model-fbec82-kserve-566f67c557-f9g2v\" (UID: \"7037c73c-59cf-4f54-a054-201b076b62cb\") " pod="ai-tenant-e2e-shared-b-w2-fbec82/auth-test-model-fbec82-kserve-566f67c557-f9g2v" Nov 02 23:43:50.025818 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:50.023832 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7037c73c-59cf-4f54-a054-201b076b62cb-tmp-dir\") pod \"auth-test-model-fbec82-kserve-566f67c557-f9g2v\" (UID: \"7037c73c-59cf-4f54-a054-201b076b62cb\") " pod="ai-tenant-e2e-shared-b-w2-fbec82/auth-test-model-fbec82-kserve-566f67c557-f9g2v" Nov 02 23:43:50.026596 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:50.026572 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7037c73c-59cf-4f54-a054-201b076b62cb-model-cache\") pod \"auth-test-model-fbec82-kserve-566f67c557-f9g2v\" (UID: \"7037c73c-59cf-4f54-a054-201b076b62cb\") " pod="ai-tenant-e2e-shared-b-w2-fbec82/auth-test-model-fbec82-kserve-566f67c557-f9g2v" Nov 02 23:43:50.032819 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:50.031760 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7037c73c-59cf-4f54-a054-201b076b62cb-dshm\") pod \"auth-test-model-fbec82-kserve-566f67c557-f9g2v\" (UID: \"7037c73c-59cf-4f54-a054-201b076b62cb\") " pod="ai-tenant-e2e-shared-b-w2-fbec82/auth-test-model-fbec82-kserve-566f67c557-f9g2v" Nov 02 23:43:50.034699 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:50.034669 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7037c73c-59cf-4f54-a054-201b076b62cb-tls-certs\") pod \"auth-test-model-fbec82-kserve-566f67c557-f9g2v\" (UID: \"7037c73c-59cf-4f54-a054-201b076b62cb\") " pod="ai-tenant-e2e-shared-b-w2-fbec82/auth-test-model-fbec82-kserve-566f67c557-f9g2v" Nov 02 23:43:50.075884 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:50.075851 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzdm6\" (UniqueName: \"kubernetes.io/projected/7037c73c-59cf-4f54-a054-201b076b62cb-kube-api-access-kzdm6\") pod \"auth-test-model-fbec82-kserve-566f67c557-f9g2v\" (UID: \"7037c73c-59cf-4f54-a054-201b076b62cb\") " pod="ai-tenant-e2e-shared-b-w2-fbec82/auth-test-model-fbec82-kserve-566f67c557-f9g2v" Nov 02 23:43:50.152728 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:50.152644 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-fbec82/auth-test-model-fbec82-kserve-566f67c557-f9g2v" Nov 02 23:43:50.520768 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:50.520738 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-fbec82/auth-test-model-fbec82-kserve-566f67c557-f9g2v"] Nov 02 23:43:50.521582 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:43:50.521559 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7037c73c_59cf_4f54_a054_201b076b62cb.slice/crio-d61c09c69b10705932cd9a34a96bf04bf45121f12a29abd2827db70dff1287ae WatchSource:0}: Error finding container d61c09c69b10705932cd9a34a96bf04bf45121f12a29abd2827db70dff1287ae: Status 404 returned error can't find the container with id d61c09c69b10705932cd9a34a96bf04bf45121f12a29abd2827db70dff1287ae Nov 02 23:43:51.491139 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:51.491104 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-fbec82/auth-test-model-fbec82-kserve-566f67c557-f9g2v" event={"ID":"7037c73c-59cf-4f54-a054-201b076b62cb","Type":"ContainerStarted","Data":"9cd4571cefabf36b85fae1db78f28cc9b46a92527e037607b5e19a1f733e35ed"} Nov 02 23:43:51.491640 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:51.491147 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-fbec82/auth-test-model-fbec82-kserve-566f67c557-f9g2v" event={"ID":"7037c73c-59cf-4f54-a054-201b076b62cb","Type":"ContainerStarted","Data":"d61c09c69b10705932cd9a34a96bf04bf45121f12a29abd2827db70dff1287ae"} Nov 02 23:43:51.600366 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:51.600306 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-fbec82/auth-test-model-fbec82-kserve-566f67c557-f9g2v" podStartSLOduration=2.600292493 podStartE2EDuration="2.600292493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:43:51.600279736 +0000 UTC m=+3080.568811522" watchObservedRunningTime="2026-11-02 23:43:51.600292493 +0000 UTC m=+3080.568824277" Nov 02 23:43:52.153204 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:52.153169 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-fbec82/auth-test-model-fbec82-kserve-566f67c557-f9g2v" Nov 02 23:43:54.312379 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:54.312332 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr"] Nov 02 23:43:54.312859 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:54.312615 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" podUID="74a2e3c3-353b-474d-83d4-c3ecc92d4963" containerName="istio-proxy" containerID="cri-o://b4b7c00de6bd9fd7e010a98629a176f99ded52e69f3061fb7b659154b94f8900" gracePeriod=30 Nov 02 23:43:55.702864 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:55.702821 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-28f82c/auth-test-model-28f82c-kserve-65859bc8d9-vdlv4"] Nov 02 23:43:55.707362 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:55.707331 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-28f82c/auth-test-model-28f82c-kserve-65859bc8d9-vdlv4" Nov 02 23:43:55.762022 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:55.761986 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-28f82c/auth-test-model-28f82c-kserve-65859bc8d9-vdlv4"] Nov 02 23:43:55.796983 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:55.796949 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20d23100-5ee8-40f1-b654-00755c3ab573-home\") pod \"auth-test-model-28f82c-kserve-65859bc8d9-vdlv4\" (UID: \"20d23100-5ee8-40f1-b654-00755c3ab573\") " pod="ai-tenant-e2e-shared-a-w3-28f82c/auth-test-model-28f82c-kserve-65859bc8d9-vdlv4" Nov 02 23:43:55.796983 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:55.796991 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20d23100-5ee8-40f1-b654-00755c3ab573-tmp-dir\") pod \"auth-test-model-28f82c-kserve-65859bc8d9-vdlv4\" (UID: \"20d23100-5ee8-40f1-b654-00755c3ab573\") " pod="ai-tenant-e2e-shared-a-w3-28f82c/auth-test-model-28f82c-kserve-65859bc8d9-vdlv4" Nov 02 23:43:55.797222 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:55.797164 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20d23100-5ee8-40f1-b654-00755c3ab573-model-cache\") pod \"auth-test-model-28f82c-kserve-65859bc8d9-vdlv4\" (UID: \"20d23100-5ee8-40f1-b654-00755c3ab573\") " pod="ai-tenant-e2e-shared-a-w3-28f82c/auth-test-model-28f82c-kserve-65859bc8d9-vdlv4" Nov 02 23:43:55.797331 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:55.797304 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdjzw\" (UniqueName: \"kubernetes.io/projected/20d23100-5ee8-40f1-b654-00755c3ab573-kube-api-access-tdjzw\") pod \"auth-test-model-28f82c-kserve-65859bc8d9-vdlv4\" (UID: \"20d23100-5ee8-40f1-b654-00755c3ab573\") " pod="ai-tenant-e2e-shared-a-w3-28f82c/auth-test-model-28f82c-kserve-65859bc8d9-vdlv4" Nov 02 23:43:55.797408 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:55.797386 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20d23100-5ee8-40f1-b654-00755c3ab573-dshm\") pod \"auth-test-model-28f82c-kserve-65859bc8d9-vdlv4\" (UID: \"20d23100-5ee8-40f1-b654-00755c3ab573\") " pod="ai-tenant-e2e-shared-a-w3-28f82c/auth-test-model-28f82c-kserve-65859bc8d9-vdlv4" Nov 02 23:43:55.797581 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:55.797561 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20d23100-5ee8-40f1-b654-00755c3ab573-tls-certs\") pod \"auth-test-model-28f82c-kserve-65859bc8d9-vdlv4\" (UID: \"20d23100-5ee8-40f1-b654-00755c3ab573\") " pod="ai-tenant-e2e-shared-a-w3-28f82c/auth-test-model-28f82c-kserve-65859bc8d9-vdlv4" Nov 02 23:43:55.804541 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:55.804516 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-28f82c\"/\"openshift-service-ca.crt\"" Nov 02 23:43:55.804835 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:55.804817 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-28f82c\"/\"default-dockercfg-2z7l9\"" Nov 02 23:43:55.804940 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:55.804913 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-28f82c\"/\"kube-root-ca.crt\"" Nov 02 23:43:55.807061 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:55.807044 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-28f82c\"/\"auth-test-model-28f82c-kserve-self-signed-certs\"" Nov 02 23:43:55.908183 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:55.898666 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20d23100-5ee8-40f1-b654-00755c3ab573-tls-certs\") pod \"auth-test-model-28f82c-kserve-65859bc8d9-vdlv4\" (UID: \"20d23100-5ee8-40f1-b654-00755c3ab573\") " pod="ai-tenant-e2e-shared-a-w3-28f82c/auth-test-model-28f82c-kserve-65859bc8d9-vdlv4" Nov 02 23:43:55.908183 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:55.898741 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20d23100-5ee8-40f1-b654-00755c3ab573-home\") pod \"auth-test-model-28f82c-kserve-65859bc8d9-vdlv4\" (UID: \"20d23100-5ee8-40f1-b654-00755c3ab573\") " pod="ai-tenant-e2e-shared-a-w3-28f82c/auth-test-model-28f82c-kserve-65859bc8d9-vdlv4" Nov 02 23:43:55.908183 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:55.898767 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20d23100-5ee8-40f1-b654-00755c3ab573-tmp-dir\") pod \"auth-test-model-28f82c-kserve-65859bc8d9-vdlv4\" (UID: \"20d23100-5ee8-40f1-b654-00755c3ab573\") " pod="ai-tenant-e2e-shared-a-w3-28f82c/auth-test-model-28f82c-kserve-65859bc8d9-vdlv4" Nov 02 23:43:55.908183 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:55.898841 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20d23100-5ee8-40f1-b654-00755c3ab573-model-cache\") pod \"auth-test-model-28f82c-kserve-65859bc8d9-vdlv4\" (UID: \"20d23100-5ee8-40f1-b654-00755c3ab573\") " pod="ai-tenant-e2e-shared-a-w3-28f82c/auth-test-model-28f82c-kserve-65859bc8d9-vdlv4" Nov 02 23:43:55.908183 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:55.898945 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdjzw\" (UniqueName: \"kubernetes.io/projected/20d23100-5ee8-40f1-b654-00755c3ab573-kube-api-access-tdjzw\") pod \"auth-test-model-28f82c-kserve-65859bc8d9-vdlv4\" (UID: \"20d23100-5ee8-40f1-b654-00755c3ab573\") " pod="ai-tenant-e2e-shared-a-w3-28f82c/auth-test-model-28f82c-kserve-65859bc8d9-vdlv4" Nov 02 23:43:55.908183 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:55.898983 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20d23100-5ee8-40f1-b654-00755c3ab573-dshm\") pod \"auth-test-model-28f82c-kserve-65859bc8d9-vdlv4\" (UID: \"20d23100-5ee8-40f1-b654-00755c3ab573\") " pod="ai-tenant-e2e-shared-a-w3-28f82c/auth-test-model-28f82c-kserve-65859bc8d9-vdlv4" Nov 02 23:43:55.908183 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:55.899796 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20d23100-5ee8-40f1-b654-00755c3ab573-tmp-dir\") pod \"auth-test-model-28f82c-kserve-65859bc8d9-vdlv4\" (UID: \"20d23100-5ee8-40f1-b654-00755c3ab573\") " pod="ai-tenant-e2e-shared-a-w3-28f82c/auth-test-model-28f82c-kserve-65859bc8d9-vdlv4" Nov 02 23:43:55.911055 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:55.910399 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20d23100-5ee8-40f1-b654-00755c3ab573-home\") pod \"auth-test-model-28f82c-kserve-65859bc8d9-vdlv4\" (UID: \"20d23100-5ee8-40f1-b654-00755c3ab573\") " pod="ai-tenant-e2e-shared-a-w3-28f82c/auth-test-model-28f82c-kserve-65859bc8d9-vdlv4" Nov 02 23:43:55.912553 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:55.911275 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20d23100-5ee8-40f1-b654-00755c3ab573-dshm\") pod \"auth-test-model-28f82c-kserve-65859bc8d9-vdlv4\" (UID: \"20d23100-5ee8-40f1-b654-00755c3ab573\") " pod="ai-tenant-e2e-shared-a-w3-28f82c/auth-test-model-28f82c-kserve-65859bc8d9-vdlv4" Nov 02 23:43:55.912553 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:55.912291 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20d23100-5ee8-40f1-b654-00755c3ab573-model-cache\") pod \"auth-test-model-28f82c-kserve-65859bc8d9-vdlv4\" (UID: \"20d23100-5ee8-40f1-b654-00755c3ab573\") " pod="ai-tenant-e2e-shared-a-w3-28f82c/auth-test-model-28f82c-kserve-65859bc8d9-vdlv4" Nov 02 23:43:55.914403 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:55.913816 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20d23100-5ee8-40f1-b654-00755c3ab573-tls-certs\") pod \"auth-test-model-28f82c-kserve-65859bc8d9-vdlv4\" (UID: \"20d23100-5ee8-40f1-b654-00755c3ab573\") " pod="ai-tenant-e2e-shared-a-w3-28f82c/auth-test-model-28f82c-kserve-65859bc8d9-vdlv4" Nov 02 23:43:55.964661 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:55.964550 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdjzw\" (UniqueName: \"kubernetes.io/projected/20d23100-5ee8-40f1-b654-00755c3ab573-kube-api-access-tdjzw\") pod \"auth-test-model-28f82c-kserve-65859bc8d9-vdlv4\" (UID: \"20d23100-5ee8-40f1-b654-00755c3ab573\") " pod="ai-tenant-e2e-shared-a-w3-28f82c/auth-test-model-28f82c-kserve-65859bc8d9-vdlv4" Nov 02 23:43:56.024936 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:56.024871 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-28f82c/auth-test-model-28f82c-kserve-65859bc8d9-vdlv4" Nov 02 23:43:56.394758 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:56.394726 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-28f82c/auth-test-model-28f82c-kserve-65859bc8d9-vdlv4"] Nov 02 23:43:56.395165 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:43:56.395140 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20d23100_5ee8_40f1_b654_00755c3ab573.slice/crio-6e0408a506d3d430f890bd40266b5f670a8cb9bd37c0d477a4a7019d2cb2a0e6 WatchSource:0}: Error finding container 6e0408a506d3d430f890bd40266b5f670a8cb9bd37c0d477a4a7019d2cb2a0e6: Status 404 returned error can't find the container with id 6e0408a506d3d430f890bd40266b5f670a8cb9bd37c0d477a4a7019d2cb2a0e6 Nov 02 23:43:56.532869 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:56.532827 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-28f82c/auth-test-model-28f82c-kserve-65859bc8d9-vdlv4" event={"ID":"20d23100-5ee8-40f1-b654-00755c3ab573","Type":"ContainerStarted","Data":"36c7db146bfcbfa7827540766343540847c6dd61fa420d95e7910ecd22d7532f"} Nov 02 23:43:56.533055 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:56.532878 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-28f82c/auth-test-model-28f82c-kserve-65859bc8d9-vdlv4" event={"ID":"20d23100-5ee8-40f1-b654-00755c3ab573","Type":"ContainerStarted","Data":"6e0408a506d3d430f890bd40266b5f670a8cb9bd37c0d477a4a7019d2cb2a0e6"} Nov 02 23:43:56.645467 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:56.645348 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-28f82c/auth-test-model-28f82c-kserve-65859bc8d9-vdlv4" podStartSLOduration=1.645324725 podStartE2EDuration="1.645324725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:43:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:43:56.64096681 +0000 UTC m=+3085.609498595" watchObservedRunningTime="2026-11-02 23:43:56.645324725 +0000 UTC m=+3085.613856511" Nov 02 23:43:57.025476 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:57.025449 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-28f82c/auth-test-model-28f82c-kserve-65859bc8d9-vdlv4" Nov 02 23:43:59.561245 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:59.561213 5076 generic.go:356] "Generic (PLEG): container finished" podID="74a2e3c3-353b-474d-83d4-c3ecc92d4963" containerID="b4b7c00de6bd9fd7e010a98629a176f99ded52e69f3061fb7b659154b94f8900" exitCode=0 Nov 02 23:43:59.561641 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:59.561283 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" event={"ID":"74a2e3c3-353b-474d-83d4-c3ecc92d4963","Type":"ContainerDied","Data":"b4b7c00de6bd9fd7e010a98629a176f99ded52e69f3061fb7b659154b94f8900"} Nov 02 23:43:59.601458 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:59.601428 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" Nov 02 23:43:59.646550 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:59.646514 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74a2e3c3-353b-474d-83d4-c3ecc92d4963-kube-api-access-rf9vn\" (UniqueName: \"kubernetes.io/projected/74a2e3c3-353b-474d-83d4-c3ecc92d4963-kube-api-access-rf9vn\") pod \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\" (UID: \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\") " Nov 02 23:43:59.646716 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:59.646583 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a2e3c3-353b-474d-83d4-c3ecc92d4963-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a2e3c3-353b-474d-83d4-c3ecc92d4963-istio-data\") pod \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\" (UID: \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\") " Nov 02 23:43:59.646716 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:59.646691 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74a2e3c3-353b-474d-83d4-c3ecc92d4963-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a2e3c3-353b-474d-83d4-c3ecc92d4963-istio-podinfo\") pod \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\" (UID: \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\") " Nov 02 23:43:59.646822 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:59.646778 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74a2e3c3-353b-474d-83d4-c3ecc92d4963-istio-token\" (UniqueName: \"kubernetes.io/projected/74a2e3c3-353b-474d-83d4-c3ecc92d4963-istio-token\") pod \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\" (UID: \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\") " Nov 02 23:43:59.646936 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:59.646832 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a2e3c3-353b-474d-83d4-c3ecc92d4963-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a2e3c3-353b-474d-83d4-c3ecc92d4963-istio-envoy\") pod \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\" (UID: \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\") " Nov 02 23:43:59.646936 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:59.646881 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a2e3c3-353b-474d-83d4-c3ecc92d4963-istio-data" pod "74a2e3c3-353b-474d-83d4-c3ecc92d4963" (UID: "74a2e3c3-353b-474d-83d4-c3ecc92d4963"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:59.646936 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:59.646925 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74a2e3c3-353b-474d-83d4-c3ecc92d4963-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a2e3c3-353b-474d-83d4-c3ecc92d4963-istiod-ca-cert\") pod \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\" (UID: \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\") " Nov 02 23:43:59.647110 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:59.646990 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a2e3c3-353b-474d-83d4-c3ecc92d4963-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a2e3c3-353b-474d-83d4-c3ecc92d4963-credential-socket\") pod \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\" (UID: \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\") " Nov 02 23:43:59.647110 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:59.647058 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a2e3c3-353b-474d-83d4-c3ecc92d4963-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a2e3c3-353b-474d-83d4-c3ecc92d4963-workload-socket\") pod \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\" (UID: \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\") " Nov 02 23:43:59.647214 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:59.647115 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a2e3c3-353b-474d-83d4-c3ecc92d4963-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a2e3c3-353b-474d-83d4-c3ecc92d4963-workload-certs\") pod \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\" (UID: \"74a2e3c3-353b-474d-83d4-c3ecc92d4963\") " Nov 02 23:43:59.647314 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:59.647283 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a2e3c3-353b-474d-83d4-c3ecc92d4963-credential-socket" pod "74a2e3c3-353b-474d-83d4-c3ecc92d4963" (UID: "74a2e3c3-353b-474d-83d4-c3ecc92d4963"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:59.647379 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:59.647305 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a2e3c3-353b-474d-83d4-c3ecc92d4963-workload-socket" pod "74a2e3c3-353b-474d-83d4-c3ecc92d4963" (UID: "74a2e3c3-353b-474d-83d4-c3ecc92d4963"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:59.647379 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:59.647296 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74a2e3c3-353b-474d-83d4-c3ecc92d4963-istiod-ca-cert" pod "74a2e3c3-353b-474d-83d4-c3ecc92d4963" (UID: "74a2e3c3-353b-474d-83d4-c3ecc92d4963"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:43:59.647567 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:59.647538 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a2e3c3-353b-474d-83d4-c3ecc92d4963-workload-certs" pod "74a2e3c3-353b-474d-83d4-c3ecc92d4963" (UID: "74a2e3c3-353b-474d-83d4-c3ecc92d4963"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:59.647725 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:59.647708 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a2e3c3-353b-474d-83d4-c3ecc92d4963-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:59.647771 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:59.647734 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a2e3c3-353b-474d-83d4-c3ecc92d4963-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:59.647771 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:59.647749 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a2e3c3-353b-474d-83d4-c3ecc92d4963-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:59.647771 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:59.647763 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a2e3c3-353b-474d-83d4-c3ecc92d4963-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:59.647866 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:59.647776 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a2e3c3-353b-474d-83d4-c3ecc92d4963-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:59.649111 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:59.649074 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a2e3c3-353b-474d-83d4-c3ecc92d4963-istio-envoy" pod "74a2e3c3-353b-474d-83d4-c3ecc92d4963" (UID: "74a2e3c3-353b-474d-83d4-c3ecc92d4963"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:59.649269 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:59.649249 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a2e3c3-353b-474d-83d4-c3ecc92d4963-kube-api-access-rf9vn" pod "74a2e3c3-353b-474d-83d4-c3ecc92d4963" (UID: "74a2e3c3-353b-474d-83d4-c3ecc92d4963"). InnerVolumeSpecName "kube-api-access-rf9vn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:43:59.649386 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:59.649365 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74a2e3c3-353b-474d-83d4-c3ecc92d4963-istio-podinfo" pod "74a2e3c3-353b-474d-83d4-c3ecc92d4963" (UID: "74a2e3c3-353b-474d-83d4-c3ecc92d4963"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:43:59.649433 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:59.649390 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a2e3c3-353b-474d-83d4-c3ecc92d4963-istio-token" pod "74a2e3c3-353b-474d-83d4-c3ecc92d4963" (UID: "74a2e3c3-353b-474d-83d4-c3ecc92d4963"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:43:59.749493 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:59.749457 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a2e3c3-353b-474d-83d4-c3ecc92d4963-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:59.749493 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:59.749489 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a2e3c3-353b-474d-83d4-c3ecc92d4963-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:59.749692 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:59.749503 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf9vn\" (UniqueName: \"kubernetes.io/projected/74a2e3c3-353b-474d-83d4-c3ecc92d4963-kube-api-access-rf9vn\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:59.749692 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:43:59.749514 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a2e3c3-353b-474d-83d4-c3ecc92d4963-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:44:00.153818 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:00.153781 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-fbec82/auth-test-model-fbec82-kserve-566f67c557-f9g2v" Nov 02 23:44:00.166709 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:00.166682 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-fbec82/auth-test-model-fbec82-kserve-566f67c557-f9g2v" Nov 02 23:44:00.568634 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:00.568592 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" event={"ID":"74a2e3c3-353b-474d-83d4-c3ecc92d4963","Type":"ContainerDied","Data":"f8bb56b2af7fe491ca3ddec67f107176ffe907a67a9aa39b3dbda7ca8c7cdea5"} Nov 02 23:44:00.568634 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:00.568616 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr" Nov 02 23:44:00.569190 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:00.568657 5076 scope.go:122] "RemoveContainer" containerID="b4b7c00de6bd9fd7e010a98629a176f99ded52e69f3061fb7b659154b94f8900" Nov 02 23:44:00.581246 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:00.581223 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-fbec82/auth-test-model-fbec82-kserve-566f67c557-f9g2v" Nov 02 23:44:00.727797 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:00.727759 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr"] Nov 02 23:44:00.775080 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:00.775039 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-bb8jr"] Nov 02 23:44:01.149517 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:01.149479 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74a2e3c3-353b-474d-83d4-c3ecc92d4963" path="/var/lib/kubelet/pods/74a2e3c3-353b-474d-83d4-c3ecc92d4963/volumes" Nov 02 23:44:06.025951 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:06.025883 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-28f82c/auth-test-model-28f82c-kserve-65859bc8d9-vdlv4" Nov 02 23:44:06.039066 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:06.039040 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-28f82c/auth-test-model-28f82c-kserve-65859bc8d9-vdlv4" Nov 02 23:44:06.618867 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:06.618832 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-28f82c/auth-test-model-28f82c-kserve-65859bc8d9-vdlv4" Nov 02 23:44:09.883988 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:44:09.883948 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:44:20.782846 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:20.782764 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8"] Nov 02 23:44:20.783348 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:20.783333 5076 cpu_manager.go:414] "containerMap: removing container" podUID="74a2e3c3-353b-474d-83d4-c3ecc92d4963" containerName="istio-proxy" Nov 02 23:44:20.783396 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:20.783348 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74a2e3c3-353b-474d-83d4-c3ecc92d4963" containerName="istio-proxy" Nov 02 23:44:20.783460 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:20.783450 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="74a2e3c3-353b-474d-83d4-c3ecc92d4963" containerName="istio-proxy" Nov 02 23:44:20.788161 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:20.788144 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8" Nov 02 23:44:20.840734 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:20.840704 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8"] Nov 02 23:44:20.883942 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:20.883911 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-52dd59-serving-cert\"" Nov 02 23:44:20.916216 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:20.916177 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0839a019-2399-45ba-8b53-439cc43e28f4-maas-api-tls\") pod \"maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8\" (UID: \"0839a019-2399-45ba-8b53-439cc43e28f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8" Nov 02 23:44:20.916415 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:20.916226 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0839a019-2399-45ba-8b53-439cc43e28f4-openshift-service-ca\") pod \"maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8\" (UID: \"0839a019-2399-45ba-8b53-439cc43e28f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8" Nov 02 23:44:20.916415 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:20.916258 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0839a019-2399-45ba-8b53-439cc43e28f4-metrics-certs\") pod \"maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8\" (UID: \"0839a019-2399-45ba-8b53-439cc43e28f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8" Nov 02 23:44:20.916415 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:20.916376 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x42rf\" (UniqueName: \"kubernetes.io/projected/0839a019-2399-45ba-8b53-439cc43e28f4-kube-api-access-x42rf\") pod \"maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8\" (UID: \"0839a019-2399-45ba-8b53-439cc43e28f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8" Nov 02 23:44:21.017521 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:21.017477 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0839a019-2399-45ba-8b53-439cc43e28f4-maas-api-tls\") pod \"maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8\" (UID: \"0839a019-2399-45ba-8b53-439cc43e28f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8" Nov 02 23:44:21.017521 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:21.017519 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0839a019-2399-45ba-8b53-439cc43e28f4-openshift-service-ca\") pod \"maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8\" (UID: \"0839a019-2399-45ba-8b53-439cc43e28f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8" Nov 02 23:44:21.017793 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:21.017539 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0839a019-2399-45ba-8b53-439cc43e28f4-metrics-certs\") pod \"maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8\" (UID: \"0839a019-2399-45ba-8b53-439cc43e28f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8" Nov 02 23:44:21.017793 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:21.017612 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x42rf\" (UniqueName: \"kubernetes.io/projected/0839a019-2399-45ba-8b53-439cc43e28f4-kube-api-access-x42rf\") pod \"maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8\" (UID: \"0839a019-2399-45ba-8b53-439cc43e28f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8" Nov 02 23:44:21.018258 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:21.018231 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0839a019-2399-45ba-8b53-439cc43e28f4-openshift-service-ca\") pod \"maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8\" (UID: \"0839a019-2399-45ba-8b53-439cc43e28f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8" Nov 02 23:44:21.020093 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:21.020071 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0839a019-2399-45ba-8b53-439cc43e28f4-metrics-certs\") pod \"maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8\" (UID: \"0839a019-2399-45ba-8b53-439cc43e28f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8" Nov 02 23:44:21.020181 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:21.020143 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0839a019-2399-45ba-8b53-439cc43e28f4-maas-api-tls\") pod \"maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8\" (UID: \"0839a019-2399-45ba-8b53-439cc43e28f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8" Nov 02 23:44:21.074779 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:21.074708 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x42rf\" (UniqueName: \"kubernetes.io/projected/0839a019-2399-45ba-8b53-439cc43e28f4-kube-api-access-x42rf\") pod \"maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8\" (UID: \"0839a019-2399-45ba-8b53-439cc43e28f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8" Nov 02 23:44:21.101507 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:21.101474 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8" Nov 02 23:44:21.476204 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:21.476173 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8"] Nov 02 23:44:21.476641 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:44:21.476617 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0839a019_2399_45ba_8b53_439cc43e28f4.slice/crio-c8bd7e7471eca9f1064eb7cfaca0cc2d26218bbf4aa04f114060ebe84a7d581b WatchSource:0}: Error finding container c8bd7e7471eca9f1064eb7cfaca0cc2d26218bbf4aa04f114060ebe84a7d581b: Status 404 returned error can't find the container with id c8bd7e7471eca9f1064eb7cfaca0cc2d26218bbf4aa04f114060ebe84a7d581b Nov 02 23:44:21.478000 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:21.477980 5076 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 23:44:21.725352 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:21.725260 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8" event={"ID":"0839a019-2399-45ba-8b53-439cc43e28f4","Type":"ContainerStarted","Data":"c8bd7e7471eca9f1064eb7cfaca0cc2d26218bbf4aa04f114060ebe84a7d581b"} Nov 02 23:44:22.735950 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:22.735916 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8" event={"ID":"0839a019-2399-45ba-8b53-439cc43e28f4","Type":"ContainerStarted","Data":"4a97cca210e6d00e250212176831e01c52a7a2d56b7dec563dadf604347e8428"} Nov 02 23:44:22.736342 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:22.736039 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8" Nov 02 23:44:22.846836 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:22.846741 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8" podStartSLOduration=1.813129304 podStartE2EDuration="2.846729466s" totalImagesPullingTime="1.033600162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:44:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:44:22.844578297 +0000 UTC m=+3111.813110081" watchObservedRunningTime="2026-11-02 23:44:22.846729466 +0000 UTC m=+3111.815261249" Nov 02 23:44:28.787864 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:28.787832 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8" Nov 02 23:44:39.886171 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:44:39.886130 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:44:45.041997 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:45.040612 5076 patch_prober.go:29] interesting pod/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": read tcp 10.132.0.2:38684->10.132.0.169:15021: read: connection reset by peer" start-of-body= Nov 02 23:44:45.041997 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:45.040665 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" podUID="ae04339e-7f59-4edd-862a-aa88108d59bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": read tcp 10.132.0.2:38684->10.132.0.169:15021: read: connection reset by peer" Nov 02 23:44:45.921729 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:45.921692 5076 generic.go:356] "Generic (PLEG): container finished" podID="ae04339e-7f59-4edd-862a-aa88108d59bb" containerID="c3aeb1cc41a839182b6d2a514319eddfd8d4c5d26c0ce431bf6e568d14dd1e9f" exitCode=0 Nov 02 23:44:45.921951 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:45.921754 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" event={"ID":"ae04339e-7f59-4edd-862a-aa88108d59bb","Type":"ContainerDied","Data":"c3aeb1cc41a839182b6d2a514319eddfd8d4c5d26c0ce431bf6e568d14dd1e9f"} Nov 02 23:44:45.922195 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:45.922181 5076 scope.go:122] "RemoveContainer" containerID="c3aeb1cc41a839182b6d2a514319eddfd8d4c5d26c0ce431bf6e568d14dd1e9f" Nov 02 23:44:45.923767 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:45.923741 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:44:45.923836 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:45.923807 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:44:45.923873 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:45.923839 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:44:46.949328 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:46.949288 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" event={"ID":"ae04339e-7f59-4edd-862a-aa88108d59bb","Type":"ContainerStarted","Data":"de5abe07d065216187bfb783e583436b6e6138d3cdb464f1b63b7580ffbe0eee"} Nov 02 23:44:46.978119 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:46.978085 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" Nov 02 23:44:46.978315 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:46.978289 5076 patch_prober.go:29] interesting pod/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 02 23:44:46.978399 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:46.978352 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" podUID="ae04339e-7f59-4edd-862a-aa88108d59bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 02 23:44:47.978235 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:47.978201 5076 patch_prober.go:29] interesting pod/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 02 23:44:47.978647 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:47.978283 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" podUID="ae04339e-7f59-4edd-862a-aa88108d59bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 02 23:44:48.978461 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:48.978425 5076 patch_prober.go:29] interesting pod/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 02 23:44:48.978866 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:48.978491 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" podUID="ae04339e-7f59-4edd-862a-aa88108d59bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 02 23:44:49.977883 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:49.977850 5076 patch_prober.go:29] interesting pod/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 02 23:44:49.978100 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:49.977931 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" podUID="ae04339e-7f59-4edd-862a-aa88108d59bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 02 23:44:50.977751 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:50.977719 5076 patch_prober.go:29] interesting pod/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 02 23:44:50.978312 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:50.977789 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" podUID="ae04339e-7f59-4edd-862a-aa88108d59bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 02 23:44:51.977703 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:51.977671 5076 patch_prober.go:29] interesting pod/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 02 23:44:51.977992 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:51.977732 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" podUID="ae04339e-7f59-4edd-862a-aa88108d59bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 02 23:44:52.982417 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:52.982382 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" Nov 02 23:44:52.982861 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:52.982790 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" Nov 02 23:44:52.983570 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:44:52.983549 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" Nov 02 23:45:00.208538 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:00.208504 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52dd59-29894385-x5c2f"] Nov 02 23:45:00.213774 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:00.213755 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52dd59-29894385-x5c2f" Nov 02 23:45:00.253031 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:00.253001 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fec430c-8002-4852-b0d2-32217cffb96b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-explicit-52dd59-29894385-x5c2f\" (UID: \"0fec430c-8002-4852-b0d2-32217cffb96b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52dd59-29894385-x5c2f" Nov 02 23:45:00.253172 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:00.253062 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zsgc\" (UniqueName: \"kubernetes.io/projected/0fec430c-8002-4852-b0d2-32217cffb96b-kube-api-access-4zsgc\") pod \"maas-api-key-cleanup-e2e-explicit-52dd59-29894385-x5c2f\" (UID: \"0fec430c-8002-4852-b0d2-32217cffb96b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52dd59-29894385-x5c2f" Nov 02 23:45:00.307385 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:00.307334 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52dd59-29894385-x5c2f"] Nov 02 23:45:00.351457 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:00.351425 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:45:00.354288 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:00.354258 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fec430c-8002-4852-b0d2-32217cffb96b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-explicit-52dd59-29894385-x5c2f\" (UID: \"0fec430c-8002-4852-b0d2-32217cffb96b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52dd59-29894385-x5c2f" Nov 02 23:45:00.354477 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:00.354354 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zsgc\" (UniqueName: \"kubernetes.io/projected/0fec430c-8002-4852-b0d2-32217cffb96b-kube-api-access-4zsgc\") pod \"maas-api-key-cleanup-e2e-explicit-52dd59-29894385-x5c2f\" (UID: \"0fec430c-8002-4852-b0d2-32217cffb96b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52dd59-29894385-x5c2f" Nov 02 23:45:00.355185 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:00.355162 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fec430c-8002-4852-b0d2-32217cffb96b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-explicit-52dd59-29894385-x5c2f\" (UID: \"0fec430c-8002-4852-b0d2-32217cffb96b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52dd59-29894385-x5c2f" Nov 02 23:45:00.412513 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:00.412477 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zsgc\" (UniqueName: \"kubernetes.io/projected/0fec430c-8002-4852-b0d2-32217cffb96b-kube-api-access-4zsgc\") pod \"maas-api-key-cleanup-e2e-explicit-52dd59-29894385-x5c2f\" (UID: \"0fec430c-8002-4852-b0d2-32217cffb96b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52dd59-29894385-x5c2f" Nov 02 23:45:00.527623 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:00.527540 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52dd59-29894385-x5c2f" Nov 02 23:45:00.893270 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:00.893241 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52dd59-29894385-x5c2f"] Nov 02 23:45:00.894188 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:45:00.894162 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fec430c_8002_4852_b0d2_32217cffb96b.slice/crio-db614e7323cb40cc77d8cd15a83de4a44fdb559344e284efa3f38fe68c830bb8 WatchSource:0}: Error finding container db614e7323cb40cc77d8cd15a83de4a44fdb559344e284efa3f38fe68c830bb8: Status 404 returned error can't find the container with id db614e7323cb40cc77d8cd15a83de4a44fdb559344e284efa3f38fe68c830bb8 Nov 02 23:45:01.033107 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:01.033072 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52dd59-29894385-x5c2f" event={"ID":"0fec430c-8002-4852-b0d2-32217cffb96b","Type":"ContainerStarted","Data":"db614e7323cb40cc77d8cd15a83de4a44fdb559344e284efa3f38fe68c830bb8"} Nov 02 23:45:02.040622 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:02.040531 5076 generic.go:356] "Generic (PLEG): container finished" podID="0fec430c-8002-4852-b0d2-32217cffb96b" containerID="d6812209b30b6841d6e2a018113429ec3fe6ab256de6c955213949d2437a9d39" exitCode=60 Nov 02 23:45:02.040622 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:02.040608 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52dd59-29894385-x5c2f" event={"ID":"0fec430c-8002-4852-b0d2-32217cffb96b","Type":"ContainerDied","Data":"d6812209b30b6841d6e2a018113429ec3fe6ab256de6c955213949d2437a9d39"} Nov 02 23:45:02.041104 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:02.040949 5076 scope.go:122] "RemoveContainer" containerID="d6812209b30b6841d6e2a018113429ec3fe6ab256de6c955213949d2437a9d39" Nov 02 23:45:03.048571 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:03.048479 5076 generic.go:356] "Generic (PLEG): container finished" podID="0fec430c-8002-4852-b0d2-32217cffb96b" containerID="da8b4d091ce967b8751afd8dd7d3578c77e7ba8a6ddb86fdc68daf5327e097ff" exitCode=60 Nov 02 23:45:03.049042 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:03.048567 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52dd59-29894385-x5c2f" event={"ID":"0fec430c-8002-4852-b0d2-32217cffb96b","Type":"ContainerDied","Data":"da8b4d091ce967b8751afd8dd7d3578c77e7ba8a6ddb86fdc68daf5327e097ff"} Nov 02 23:45:03.049042 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:03.048635 5076 scope.go:122] "RemoveContainer" containerID="d6812209b30b6841d6e2a018113429ec3fe6ab256de6c955213949d2437a9d39" Nov 02 23:45:03.049042 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:03.048943 5076 scope.go:122] "RemoveContainer" containerID="da8b4d091ce967b8751afd8dd7d3578c77e7ba8a6ddb86fdc68daf5327e097ff" Nov 02 23:45:03.049286 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:45:03.049267 5076 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-explicit-52dd59-29894385-x5c2f_odh-ai-gateway-infra(0fec430c-8002-4852-b0d2-32217cffb96b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52dd59-29894385-x5c2f" podUID="0fec430c-8002-4852-b0d2-32217cffb96b" Nov 02 23:45:04.057795 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:04.057767 5076 scope.go:122] "RemoveContainer" containerID="da8b4d091ce967b8751afd8dd7d3578c77e7ba8a6ddb86fdc68daf5327e097ff" Nov 02 23:45:04.058278 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:45:04.058119 5076 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-explicit-52dd59-29894385-x5c2f_odh-ai-gateway-infra(0fec430c-8002-4852-b0d2-32217cffb96b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52dd59-29894385-x5c2f" podUID="0fec430c-8002-4852-b0d2-32217cffb96b" Nov 02 23:45:09.886861 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:45:09.886821 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:45:16.141141 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:16.141112 5076 scope.go:122] "RemoveContainer" containerID="da8b4d091ce967b8751afd8dd7d3578c77e7ba8a6ddb86fdc68daf5327e097ff" Nov 02 23:45:17.139229 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:17.139193 5076 generic.go:356] "Generic (PLEG): container finished" podID="0fec430c-8002-4852-b0d2-32217cffb96b" containerID="80a91872a8165c3d7a80881013043e9aec6fbb25ad7e035b2639d8eb71f4bf84" exitCode=60 Nov 02 23:45:17.139378 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:17.139266 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52dd59-29894385-x5c2f" event={"ID":"0fec430c-8002-4852-b0d2-32217cffb96b","Type":"ContainerDied","Data":"80a91872a8165c3d7a80881013043e9aec6fbb25ad7e035b2639d8eb71f4bf84"} Nov 02 23:45:17.139378 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:17.139325 5076 scope.go:122] "RemoveContainer" containerID="da8b4d091ce967b8751afd8dd7d3578c77e7ba8a6ddb86fdc68daf5327e097ff" Nov 02 23:45:17.139823 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:17.139802 5076 scope.go:122] "RemoveContainer" containerID="80a91872a8165c3d7a80881013043e9aec6fbb25ad7e035b2639d8eb71f4bf84" Nov 02 23:45:17.140209 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:45:17.140183 5076 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-explicit-52dd59-29894385-x5c2f_odh-ai-gateway-infra(0fec430c-8002-4852-b0d2-32217cffb96b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52dd59-29894385-x5c2f" podUID="0fec430c-8002-4852-b0d2-32217cffb96b" Nov 02 23:45:18.260052 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:18.260015 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52dd59-29894385-x5c2f"] Nov 02 23:45:18.429766 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:18.429739 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52dd59-29894385-x5c2f" Nov 02 23:45:18.504273 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:18.504237 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fec430c-8002-4852-b0d2-32217cffb96b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fec430c-8002-4852-b0d2-32217cffb96b-maas-api-service-ca\") pod \"0fec430c-8002-4852-b0d2-32217cffb96b\" (UID: \"0fec430c-8002-4852-b0d2-32217cffb96b\") " Nov 02 23:45:18.504449 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:18.504410 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fec430c-8002-4852-b0d2-32217cffb96b-kube-api-access-4zsgc\" (UniqueName: \"kubernetes.io/projected/0fec430c-8002-4852-b0d2-32217cffb96b-kube-api-access-4zsgc\") pod \"0fec430c-8002-4852-b0d2-32217cffb96b\" (UID: \"0fec430c-8002-4852-b0d2-32217cffb96b\") " Nov 02 23:45:18.504842 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:18.504637 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fec430c-8002-4852-b0d2-32217cffb96b-maas-api-service-ca" pod "0fec430c-8002-4852-b0d2-32217cffb96b" (UID: "0fec430c-8002-4852-b0d2-32217cffb96b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:45:18.516071 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:18.515984 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fec430c-8002-4852-b0d2-32217cffb96b-kube-api-access-4zsgc" pod "0fec430c-8002-4852-b0d2-32217cffb96b" (UID: "0fec430c-8002-4852-b0d2-32217cffb96b"). InnerVolumeSpecName "kube-api-access-4zsgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:45:18.606763 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:18.606722 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zsgc\" (UniqueName: \"kubernetes.io/projected/0fec430c-8002-4852-b0d2-32217cffb96b-kube-api-access-4zsgc\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:18.606763 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:18.606762 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fec430c-8002-4852-b0d2-32217cffb96b-maas-api-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:19.154652 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:19.154621 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52dd59-29894385-x5c2f" Nov 02 23:45:19.154652 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:19.154642 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52dd59-29894385-x5c2f" event={"ID":"0fec430c-8002-4852-b0d2-32217cffb96b","Type":"ContainerDied","Data":"db614e7323cb40cc77d8cd15a83de4a44fdb559344e284efa3f38fe68c830bb8"} Nov 02 23:45:19.154912 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:19.154703 5076 scope.go:122] "RemoveContainer" containerID="80a91872a8165c3d7a80881013043e9aec6fbb25ad7e035b2639d8eb71f4bf84" Nov 02 23:45:19.312049 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:19.311984 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52dd59-29894385-x5c2f"] Nov 02 23:45:19.316359 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:19.316329 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-52dd59-29894385-x5c2f"] Nov 02 23:45:21.148702 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:45:21.148668 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fec430c-8002-4852-b0d2-32217cffb96b" path="/var/lib/kubelet/pods/0fec430c-8002-4852-b0d2-32217cffb96b/volumes" Nov 02 23:45:39.884688 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:45:39.884651 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:46:09.884637 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:46:09.884562 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:46:39.883753 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:46:39.883703 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:47:09.883121 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:47:09.883072 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:47:31.934473 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:31.934445 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 02 23:47:31.961177 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:31.961152 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 02 23:47:39.883635 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:47:39.883595 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:47:46.958274 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:46.958228 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8"] Nov 02 23:47:46.958855 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:46.958482 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8" podUID="0839a019-2399-45ba-8b53-439cc43e28f4" containerName="maas-api" containerID="cri-o://4a97cca210e6d00e250212176831e01c52a7a2d56b7dec563dadf604347e8428" gracePeriod=30 Nov 02 23:47:47.109050 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:47.109015 5076 generic.go:356] "Generic (PLEG): container finished" podID="0839a019-2399-45ba-8b53-439cc43e28f4" containerID="4a97cca210e6d00e250212176831e01c52a7a2d56b7dec563dadf604347e8428" exitCode=0 Nov 02 23:47:47.109218 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:47.109085 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8" event={"ID":"0839a019-2399-45ba-8b53-439cc43e28f4","Type":"ContainerDied","Data":"4a97cca210e6d00e250212176831e01c52a7a2d56b7dec563dadf604347e8428"} Nov 02 23:47:47.265983 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:47.265957 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8" Nov 02 23:47:47.307265 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:47.307232 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0839a019-2399-45ba-8b53-439cc43e28f4-kube-api-access-x42rf\" (UniqueName: \"kubernetes.io/projected/0839a019-2399-45ba-8b53-439cc43e28f4-kube-api-access-x42rf\") pod \"0839a019-2399-45ba-8b53-439cc43e28f4\" (UID: \"0839a019-2399-45ba-8b53-439cc43e28f4\") " Nov 02 23:47:47.307443 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:47.307427 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0839a019-2399-45ba-8b53-439cc43e28f4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0839a019-2399-45ba-8b53-439cc43e28f4-metrics-certs\") pod \"0839a019-2399-45ba-8b53-439cc43e28f4\" (UID: \"0839a019-2399-45ba-8b53-439cc43e28f4\") " Nov 02 23:47:47.307489 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:47.307469 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0839a019-2399-45ba-8b53-439cc43e28f4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0839a019-2399-45ba-8b53-439cc43e28f4-maas-api-tls\") pod \"0839a019-2399-45ba-8b53-439cc43e28f4\" (UID: \"0839a019-2399-45ba-8b53-439cc43e28f4\") " Nov 02 23:47:47.307603 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:47.307589 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0839a019-2399-45ba-8b53-439cc43e28f4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0839a019-2399-45ba-8b53-439cc43e28f4-openshift-service-ca\") pod \"0839a019-2399-45ba-8b53-439cc43e28f4\" (UID: \"0839a019-2399-45ba-8b53-439cc43e28f4\") " Nov 02 23:47:47.308032 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:47.308003 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0839a019-2399-45ba-8b53-439cc43e28f4-openshift-service-ca" pod "0839a019-2399-45ba-8b53-439cc43e28f4" (UID: "0839a019-2399-45ba-8b53-439cc43e28f4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:47:47.309360 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:47.309332 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0839a019-2399-45ba-8b53-439cc43e28f4-kube-api-access-x42rf" pod "0839a019-2399-45ba-8b53-439cc43e28f4" (UID: "0839a019-2399-45ba-8b53-439cc43e28f4"). InnerVolumeSpecName "kube-api-access-x42rf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:47:47.309478 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:47.309450 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0839a019-2399-45ba-8b53-439cc43e28f4-metrics-certs" pod "0839a019-2399-45ba-8b53-439cc43e28f4" (UID: "0839a019-2399-45ba-8b53-439cc43e28f4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:47:47.309523 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:47.309498 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0839a019-2399-45ba-8b53-439cc43e28f4-maas-api-tls" pod "0839a019-2399-45ba-8b53-439cc43e28f4" (UID: "0839a019-2399-45ba-8b53-439cc43e28f4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:47:47.408989 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:47.408957 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0839a019-2399-45ba-8b53-439cc43e28f4-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:47:47.408989 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:47.408988 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x42rf\" (UniqueName: \"kubernetes.io/projected/0839a019-2399-45ba-8b53-439cc43e28f4-kube-api-access-x42rf\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:47:47.409281 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:47.409003 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0839a019-2399-45ba-8b53-439cc43e28f4-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:47:47.409281 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:47.409015 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0839a019-2399-45ba-8b53-439cc43e28f4-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:47:48.116614 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:48.116589 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8" Nov 02 23:47:48.117096 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:48.116587 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8" event={"ID":"0839a019-2399-45ba-8b53-439cc43e28f4","Type":"ContainerDied","Data":"c8bd7e7471eca9f1064eb7cfaca0cc2d26218bbf4aa04f114060ebe84a7d581b"} Nov 02 23:47:48.117096 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:48.116746 5076 scope.go:122] "RemoveContainer" containerID="4a97cca210e6d00e250212176831e01c52a7a2d56b7dec563dadf604347e8428" Nov 02 23:47:48.274698 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:48.274666 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8"] Nov 02 23:47:48.322207 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:48.322171 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-52dd59-57ffdfd45-8c7l8"] Nov 02 23:47:49.147885 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:49.147840 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0839a019-2399-45ba-8b53-439cc43e28f4" path="/var/lib/kubelet/pods/0839a019-2399-45ba-8b53-439cc43e28f4/volumes" Nov 02 23:47:53.005055 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:53.005008 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w"] Nov 02 23:47:53.005550 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:53.005331 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" podUID="ae04339e-7f59-4edd-862a-aa88108d59bb" containerName="istio-proxy" containerID="cri-o://de5abe07d065216187bfb783e583436b6e6138d3cdb464f1b63b7580ffbe0eee" gracePeriod=30 Nov 02 23:47:54.078744 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.078710 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5"] Nov 02 23:47:54.079485 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.079462 5076 cpu_manager.go:414] "containerMap: removing container" podUID="0fec430c-8002-4852-b0d2-32217cffb96b" containerName="cleanup" Nov 02 23:47:54.079614 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.079497 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fec430c-8002-4852-b0d2-32217cffb96b" containerName="cleanup" Nov 02 23:47:54.079614 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.079535 5076 cpu_manager.go:414] "containerMap: removing container" podUID="0fec430c-8002-4852-b0d2-32217cffb96b" containerName="cleanup" Nov 02 23:47:54.079614 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.079542 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fec430c-8002-4852-b0d2-32217cffb96b" containerName="cleanup" Nov 02 23:47:54.079614 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.079554 5076 cpu_manager.go:414] "containerMap: removing container" podUID="0839a019-2399-45ba-8b53-439cc43e28f4" containerName="maas-api" Nov 02 23:47:54.079614 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.079561 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0839a019-2399-45ba-8b53-439cc43e28f4" containerName="maas-api" Nov 02 23:47:54.079614 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.079574 5076 cpu_manager.go:414] "containerMap: removing container" podUID="0fec430c-8002-4852-b0d2-32217cffb96b" containerName="cleanup" Nov 02 23:47:54.079614 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.079580 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fec430c-8002-4852-b0d2-32217cffb96b" containerName="cleanup" Nov 02 23:47:54.080016 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.079767 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fec430c-8002-4852-b0d2-32217cffb96b" containerName="cleanup" Nov 02 23:47:54.080016 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.079784 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fec430c-8002-4852-b0d2-32217cffb96b" containerName="cleanup" Nov 02 23:47:54.080016 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.079795 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="0839a019-2399-45ba-8b53-439cc43e28f4" containerName="maas-api" Nov 02 23:47:54.084730 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.084708 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" Nov 02 23:47:54.210744 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.210371 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b281dea-bfcb-4edf-a421-d1bff5f9767c-credential-socket\") pod \"e2e-noten-313523-openshift-default-6fd45f8999-ncsh5\" (UID: \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\") " pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" Nov 02 23:47:54.210744 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.210468 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b281dea-bfcb-4edf-a421-d1bff5f9767c-istio-podinfo\") pod \"e2e-noten-313523-openshift-default-6fd45f8999-ncsh5\" (UID: \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\") " pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" Nov 02 23:47:54.210744 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.210508 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b281dea-bfcb-4edf-a421-d1bff5f9767c-workload-certs\") pod \"e2e-noten-313523-openshift-default-6fd45f8999-ncsh5\" (UID: \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\") " pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" Nov 02 23:47:54.210744 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.210578 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b281dea-bfcb-4edf-a421-d1bff5f9767c-istiod-ca-cert\") pod \"e2e-noten-313523-openshift-default-6fd45f8999-ncsh5\" (UID: \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\") " pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" Nov 02 23:47:54.210744 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.210629 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b281dea-bfcb-4edf-a421-d1bff5f9767c-istio-token\") pod \"e2e-noten-313523-openshift-default-6fd45f8999-ncsh5\" (UID: \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\") " pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" Nov 02 23:47:54.210744 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.210660 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b281dea-bfcb-4edf-a421-d1bff5f9767c-istio-envoy\") pod \"e2e-noten-313523-openshift-default-6fd45f8999-ncsh5\" (UID: \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\") " pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" Nov 02 23:47:54.210744 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.210706 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b281dea-bfcb-4edf-a421-d1bff5f9767c-workload-socket\") pod \"e2e-noten-313523-openshift-default-6fd45f8999-ncsh5\" (UID: \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\") " pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" Nov 02 23:47:54.211289 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.210912 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b281dea-bfcb-4edf-a421-d1bff5f9767c-istio-data\") pod \"e2e-noten-313523-openshift-default-6fd45f8999-ncsh5\" (UID: \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\") " pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" Nov 02 23:47:54.211289 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.210952 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-955hz\" (UniqueName: \"kubernetes.io/projected/2b281dea-bfcb-4edf-a421-d1bff5f9767c-kube-api-access-955hz\") pod \"e2e-noten-313523-openshift-default-6fd45f8999-ncsh5\" (UID: \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\") " pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" Nov 02 23:47:54.216769 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.216744 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-313523-openshift-default-dockercfg-t88mj\"" Nov 02 23:47:54.217037 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.217007 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5"] Nov 02 23:47:54.311756 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.311708 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b281dea-bfcb-4edf-a421-d1bff5f9767c-istiod-ca-cert\") pod \"e2e-noten-313523-openshift-default-6fd45f8999-ncsh5\" (UID: \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\") " pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" Nov 02 23:47:54.311956 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.311874 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b281dea-bfcb-4edf-a421-d1bff5f9767c-istio-token\") pod \"e2e-noten-313523-openshift-default-6fd45f8999-ncsh5\" (UID: \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\") " pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" Nov 02 23:47:54.311956 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.311933 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b281dea-bfcb-4edf-a421-d1bff5f9767c-istio-envoy\") pod \"e2e-noten-313523-openshift-default-6fd45f8999-ncsh5\" (UID: \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\") " pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" Nov 02 23:47:54.312078 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.311978 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b281dea-bfcb-4edf-a421-d1bff5f9767c-workload-socket\") pod \"e2e-noten-313523-openshift-default-6fd45f8999-ncsh5\" (UID: \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\") " pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" Nov 02 23:47:54.312078 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.312028 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b281dea-bfcb-4edf-a421-d1bff5f9767c-istio-data\") pod \"e2e-noten-313523-openshift-default-6fd45f8999-ncsh5\" (UID: \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\") " pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" Nov 02 23:47:54.312078 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.312044 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-955hz\" (UniqueName: \"kubernetes.io/projected/2b281dea-bfcb-4edf-a421-d1bff5f9767c-kube-api-access-955hz\") pod \"e2e-noten-313523-openshift-default-6fd45f8999-ncsh5\" (UID: \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\") " pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" Nov 02 23:47:54.312265 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.312092 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b281dea-bfcb-4edf-a421-d1bff5f9767c-credential-socket\") pod \"e2e-noten-313523-openshift-default-6fd45f8999-ncsh5\" (UID: \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\") " pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" Nov 02 23:47:54.312265 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.312177 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b281dea-bfcb-4edf-a421-d1bff5f9767c-istio-podinfo\") pod \"e2e-noten-313523-openshift-default-6fd45f8999-ncsh5\" (UID: \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\") " pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" Nov 02 23:47:54.312265 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.312219 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b281dea-bfcb-4edf-a421-d1bff5f9767c-workload-certs\") pod \"e2e-noten-313523-openshift-default-6fd45f8999-ncsh5\" (UID: \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\") " pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" Nov 02 23:47:54.312490 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.312472 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b281dea-bfcb-4edf-a421-d1bff5f9767c-istio-data\") pod \"e2e-noten-313523-openshift-default-6fd45f8999-ncsh5\" (UID: \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\") " pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" Nov 02 23:47:54.312543 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.312499 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b281dea-bfcb-4edf-a421-d1bff5f9767c-istiod-ca-cert\") pod \"e2e-noten-313523-openshift-default-6fd45f8999-ncsh5\" (UID: \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\") " pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" Nov 02 23:47:54.312590 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.312567 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b281dea-bfcb-4edf-a421-d1bff5f9767c-workload-socket\") pod \"e2e-noten-313523-openshift-default-6fd45f8999-ncsh5\" (UID: \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\") " pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" Nov 02 23:47:54.312838 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.312782 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b281dea-bfcb-4edf-a421-d1bff5f9767c-credential-socket\") pod \"e2e-noten-313523-openshift-default-6fd45f8999-ncsh5\" (UID: \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\") " pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" Nov 02 23:47:54.312838 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.312821 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b281dea-bfcb-4edf-a421-d1bff5f9767c-workload-certs\") pod \"e2e-noten-313523-openshift-default-6fd45f8999-ncsh5\" (UID: \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\") " pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" Nov 02 23:47:54.314982 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.314957 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b281dea-bfcb-4edf-a421-d1bff5f9767c-istio-envoy\") pod \"e2e-noten-313523-openshift-default-6fd45f8999-ncsh5\" (UID: \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\") " pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" Nov 02 23:47:54.315193 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.315175 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b281dea-bfcb-4edf-a421-d1bff5f9767c-istio-podinfo\") pod \"e2e-noten-313523-openshift-default-6fd45f8999-ncsh5\" (UID: \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\") " pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" Nov 02 23:47:54.366875 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.366808 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b281dea-bfcb-4edf-a421-d1bff5f9767c-istio-token\") pod \"e2e-noten-313523-openshift-default-6fd45f8999-ncsh5\" (UID: \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\") " pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" Nov 02 23:47:54.367043 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.367019 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-955hz\" (UniqueName: \"kubernetes.io/projected/2b281dea-bfcb-4edf-a421-d1bff5f9767c-kube-api-access-955hz\") pod \"e2e-noten-313523-openshift-default-6fd45f8999-ncsh5\" (UID: \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\") " pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" Nov 02 23:47:54.398436 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.398397 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" Nov 02 23:47:54.819096 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.819067 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5"] Nov 02 23:47:54.820159 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:47:54.820132 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b281dea_bfcb_4edf_a421_d1bff5f9767c.slice/crio-9672127ab20da8a127f26315c619699c1f803ba9cfc97583ba868daca47473e9 WatchSource:0}: Error finding container 9672127ab20da8a127f26315c619699c1f803ba9cfc97583ba868daca47473e9: Status 404 returned error can't find the container with id 9672127ab20da8a127f26315c619699c1f803ba9cfc97583ba868daca47473e9 Nov 02 23:47:54.822626 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.822594 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:47:54.822708 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.822662 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:47:54.822708 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:54.822696 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:47:55.220072 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:55.220035 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" event={"ID":"2b281dea-bfcb-4edf-a421-d1bff5f9767c","Type":"ContainerStarted","Data":"84e2e0a98dc6dd0db04cb123307f3707aba86be25c3629301de9b4328b09b7d4"} Nov 02 23:47:55.220541 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:55.220098 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" event={"ID":"2b281dea-bfcb-4edf-a421-d1bff5f9767c","Type":"ContainerStarted","Data":"9672127ab20da8a127f26315c619699c1f803ba9cfc97583ba868daca47473e9"} Nov 02 23:47:55.332261 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:55.332206 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" podStartSLOduration=1.332190206 podStartE2EDuration="1.332190206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:47:55.330076353 +0000 UTC m=+3324.298608148" watchObservedRunningTime="2026-11-02 23:47:55.332190206 +0000 UTC m=+3324.300722035" Nov 02 23:47:55.398750 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:55.398714 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" Nov 02 23:47:55.405511 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:55.405484 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" Nov 02 23:47:56.225848 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:56.225818 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" Nov 02 23:47:56.227365 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:56.227344 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" Nov 02 23:47:58.242385 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:58.242348 5076 generic.go:356] "Generic (PLEG): container finished" podID="ae04339e-7f59-4edd-862a-aa88108d59bb" containerID="de5abe07d065216187bfb783e583436b6e6138d3cdb464f1b63b7580ffbe0eee" exitCode=0 Nov 02 23:47:58.242873 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:58.242429 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" event={"ID":"ae04339e-7f59-4edd-862a-aa88108d59bb","Type":"ContainerDied","Data":"de5abe07d065216187bfb783e583436b6e6138d3cdb464f1b63b7580ffbe0eee"} Nov 02 23:47:58.242873 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:58.242511 5076 scope.go:122] "RemoveContainer" containerID="c3aeb1cc41a839182b6d2a514319eddfd8d4c5d26c0ce431bf6e568d14dd1e9f" Nov 02 23:47:58.318630 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:58.318598 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" Nov 02 23:47:58.360315 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:58.360278 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae04339e-7f59-4edd-862a-aa88108d59bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae04339e-7f59-4edd-862a-aa88108d59bb-istiod-ca-cert\") pod \"ae04339e-7f59-4edd-862a-aa88108d59bb\" (UID: \"ae04339e-7f59-4edd-862a-aa88108d59bb\") " Nov 02 23:47:58.360496 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:58.360327 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae04339e-7f59-4edd-862a-aa88108d59bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae04339e-7f59-4edd-862a-aa88108d59bb-istio-podinfo\") pod \"ae04339e-7f59-4edd-862a-aa88108d59bb\" (UID: \"ae04339e-7f59-4edd-862a-aa88108d59bb\") " Nov 02 23:47:58.360496 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:58.360417 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae04339e-7f59-4edd-862a-aa88108d59bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae04339e-7f59-4edd-862a-aa88108d59bb-istio-envoy\") pod \"ae04339e-7f59-4edd-862a-aa88108d59bb\" (UID: \"ae04339e-7f59-4edd-862a-aa88108d59bb\") " Nov 02 23:47:58.360496 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:58.360442 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae04339e-7f59-4edd-862a-aa88108d59bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae04339e-7f59-4edd-862a-aa88108d59bb-workload-socket\") pod \"ae04339e-7f59-4edd-862a-aa88108d59bb\" (UID: \"ae04339e-7f59-4edd-862a-aa88108d59bb\") " Nov 02 23:47:58.360654 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:58.360527 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae04339e-7f59-4edd-862a-aa88108d59bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae04339e-7f59-4edd-862a-aa88108d59bb-workload-certs\") pod \"ae04339e-7f59-4edd-862a-aa88108d59bb\" (UID: \"ae04339e-7f59-4edd-862a-aa88108d59bb\") " Nov 02 23:47:58.360654 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:58.360545 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae04339e-7f59-4edd-862a-aa88108d59bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae04339e-7f59-4edd-862a-aa88108d59bb-credential-socket\") pod \"ae04339e-7f59-4edd-862a-aa88108d59bb\" (UID: \"ae04339e-7f59-4edd-862a-aa88108d59bb\") " Nov 02 23:47:58.360654 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:58.360594 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae04339e-7f59-4edd-862a-aa88108d59bb-kube-api-access-qlj2x\" (UniqueName: \"kubernetes.io/projected/ae04339e-7f59-4edd-862a-aa88108d59bb-kube-api-access-qlj2x\") pod \"ae04339e-7f59-4edd-862a-aa88108d59bb\" (UID: \"ae04339e-7f59-4edd-862a-aa88108d59bb\") " Nov 02 23:47:58.360654 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:58.360623 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae04339e-7f59-4edd-862a-aa88108d59bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae04339e-7f59-4edd-862a-aa88108d59bb-istio-data\") pod \"ae04339e-7f59-4edd-862a-aa88108d59bb\" (UID: \"ae04339e-7f59-4edd-862a-aa88108d59bb\") " Nov 02 23:47:58.360870 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:58.360671 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae04339e-7f59-4edd-862a-aa88108d59bb-istio-token\" (UniqueName: \"kubernetes.io/projected/ae04339e-7f59-4edd-862a-aa88108d59bb-istio-token\") pod \"ae04339e-7f59-4edd-862a-aa88108d59bb\" (UID: \"ae04339e-7f59-4edd-862a-aa88108d59bb\") " Nov 02 23:47:58.360870 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:58.360733 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae04339e-7f59-4edd-862a-aa88108d59bb-istiod-ca-cert" pod "ae04339e-7f59-4edd-862a-aa88108d59bb" (UID: "ae04339e-7f59-4edd-862a-aa88108d59bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:47:58.360870 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:58.360841 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae04339e-7f59-4edd-862a-aa88108d59bb-workload-socket" pod "ae04339e-7f59-4edd-862a-aa88108d59bb" (UID: "ae04339e-7f59-4edd-862a-aa88108d59bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:47:58.360870 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:58.360856 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae04339e-7f59-4edd-862a-aa88108d59bb-workload-certs" pod "ae04339e-7f59-4edd-862a-aa88108d59bb" (UID: "ae04339e-7f59-4edd-862a-aa88108d59bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:47:58.361140 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:58.361095 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae04339e-7f59-4edd-862a-aa88108d59bb-credential-socket" pod "ae04339e-7f59-4edd-862a-aa88108d59bb" (UID: "ae04339e-7f59-4edd-862a-aa88108d59bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:47:58.361140 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:58.361110 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae04339e-7f59-4edd-862a-aa88108d59bb-istio-data" pod "ae04339e-7f59-4edd-862a-aa88108d59bb" (UID: "ae04339e-7f59-4edd-862a-aa88108d59bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:47:58.361336 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:58.361299 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae04339e-7f59-4edd-862a-aa88108d59bb-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:47:58.361336 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:58.361330 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae04339e-7f59-4edd-862a-aa88108d59bb-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:47:58.361530 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:58.361344 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae04339e-7f59-4edd-862a-aa88108d59bb-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:47:58.361530 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:58.361358 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae04339e-7f59-4edd-862a-aa88108d59bb-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:47:58.361530 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:58.361375 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae04339e-7f59-4edd-862a-aa88108d59bb-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:47:58.363070 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:58.363043 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae04339e-7f59-4edd-862a-aa88108d59bb-istio-podinfo" pod "ae04339e-7f59-4edd-862a-aa88108d59bb" (UID: "ae04339e-7f59-4edd-862a-aa88108d59bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:47:58.363188 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:58.363073 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae04339e-7f59-4edd-862a-aa88108d59bb-kube-api-access-qlj2x" pod "ae04339e-7f59-4edd-862a-aa88108d59bb" (UID: "ae04339e-7f59-4edd-862a-aa88108d59bb"). InnerVolumeSpecName "kube-api-access-qlj2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:47:58.363188 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:58.363095 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae04339e-7f59-4edd-862a-aa88108d59bb-istio-token" pod "ae04339e-7f59-4edd-862a-aa88108d59bb" (UID: "ae04339e-7f59-4edd-862a-aa88108d59bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:47:58.363377 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:58.363361 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae04339e-7f59-4edd-862a-aa88108d59bb-istio-envoy" pod "ae04339e-7f59-4edd-862a-aa88108d59bb" (UID: "ae04339e-7f59-4edd-862a-aa88108d59bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:47:58.462721 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:58.462678 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae04339e-7f59-4edd-862a-aa88108d59bb-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:47:58.462721 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:58.462720 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae04339e-7f59-4edd-862a-aa88108d59bb-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:47:58.462976 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:58.462736 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlj2x\" (UniqueName: \"kubernetes.io/projected/ae04339e-7f59-4edd-862a-aa88108d59bb-kube-api-access-qlj2x\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:47:58.462976 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:58.462751 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae04339e-7f59-4edd-862a-aa88108d59bb-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:47:59.250384 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:59.250348 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" Nov 02 23:47:59.250866 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:59.250385 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w" event={"ID":"ae04339e-7f59-4edd-862a-aa88108d59bb","Type":"ContainerDied","Data":"f741551e1646c106f2237badd39369ab106fa1d595c25b62a8857a29a6756adb"} Nov 02 23:47:59.250866 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:59.250442 5076 scope.go:122] "RemoveContainer" containerID="de5abe07d065216187bfb783e583436b6e6138d3cdb464f1b63b7580ffbe0eee" Nov 02 23:47:59.456599 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:59.456565 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w"] Nov 02 23:47:59.503865 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:47:59.503828 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-52dd59-openshift-default-6d5cf86945-ckv5w"] Nov 02 23:48:01.149146 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:01.149111 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae04339e-7f59-4edd-862a-aa88108d59bb" path="/var/lib/kubelet/pods/ae04339e-7f59-4edd-862a-aa88108d59bb/volumes" Nov 02 23:48:09.884329 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:48:09.884287 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:48:39.884257 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:48:39.884218 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:48:53.774117 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:53.774039 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5"] Nov 02 23:48:53.774527 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:53.774293 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" podUID="2b281dea-bfcb-4edf-a421-d1bff5f9767c" containerName="istio-proxy" containerID="cri-o://84e2e0a98dc6dd0db04cb123307f3707aba86be25c3629301de9b4328b09b7d4" gracePeriod=30 Nov 02 23:48:56.228746 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:56.228707 5076 patch_prober.go:29] interesting pod/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:48:56.229244 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:56.228777 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" podUID="2b281dea-bfcb-4edf-a421-d1bff5f9767c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:48:59.060260 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:59.060232 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" Nov 02 23:48:59.127191 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:59.127157 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b281dea-bfcb-4edf-a421-d1bff5f9767c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b281dea-bfcb-4edf-a421-d1bff5f9767c-istiod-ca-cert\") pod \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\" (UID: \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\") " Nov 02 23:48:59.127361 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:59.127202 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b281dea-bfcb-4edf-a421-d1bff5f9767c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b281dea-bfcb-4edf-a421-d1bff5f9767c-credential-socket\") pod \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\" (UID: \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\") " Nov 02 23:48:59.127361 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:59.127320 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b281dea-bfcb-4edf-a421-d1bff5f9767c-kube-api-access-955hz\" (UniqueName: \"kubernetes.io/projected/2b281dea-bfcb-4edf-a421-d1bff5f9767c-kube-api-access-955hz\") pod \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\" (UID: \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\") " Nov 02 23:48:59.127442 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:59.127368 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b281dea-bfcb-4edf-a421-d1bff5f9767c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b281dea-bfcb-4edf-a421-d1bff5f9767c-istio-podinfo\") pod \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\" (UID: \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\") " Nov 02 23:48:59.127442 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:59.127399 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b281dea-bfcb-4edf-a421-d1bff5f9767c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b281dea-bfcb-4edf-a421-d1bff5f9767c-istio-data\") pod \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\" (UID: \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\") " Nov 02 23:48:59.127555 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:59.127529 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b281dea-bfcb-4edf-a421-d1bff5f9767c-credential-socket" pod "2b281dea-bfcb-4edf-a421-d1bff5f9767c" (UID: "2b281dea-bfcb-4edf-a421-d1bff5f9767c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:48:59.127623 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:59.127555 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b281dea-bfcb-4edf-a421-d1bff5f9767c-istio-token\" (UniqueName: \"kubernetes.io/projected/2b281dea-bfcb-4edf-a421-d1bff5f9767c-istio-token\") pod \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\" (UID: \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\") " Nov 02 23:48:59.127623 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:59.127579 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b281dea-bfcb-4edf-a421-d1bff5f9767c-istiod-ca-cert" pod "2b281dea-bfcb-4edf-a421-d1bff5f9767c" (UID: "2b281dea-bfcb-4edf-a421-d1bff5f9767c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:48:59.127623 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:59.127585 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b281dea-bfcb-4edf-a421-d1bff5f9767c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b281dea-bfcb-4edf-a421-d1bff5f9767c-istio-envoy\") pod \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\" (UID: \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\") " Nov 02 23:48:59.127788 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:59.127657 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b281dea-bfcb-4edf-a421-d1bff5f9767c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b281dea-bfcb-4edf-a421-d1bff5f9767c-workload-socket\") pod \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\" (UID: \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\") " Nov 02 23:48:59.127788 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:59.127768 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b281dea-bfcb-4edf-a421-d1bff5f9767c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b281dea-bfcb-4edf-a421-d1bff5f9767c-workload-certs\") pod \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\" (UID: \"2b281dea-bfcb-4edf-a421-d1bff5f9767c\") " Nov 02 23:48:59.127886 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:59.127790 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b281dea-bfcb-4edf-a421-d1bff5f9767c-istio-data" pod "2b281dea-bfcb-4edf-a421-d1bff5f9767c" (UID: "2b281dea-bfcb-4edf-a421-d1bff5f9767c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:48:59.127998 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:59.127971 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b281dea-bfcb-4edf-a421-d1bff5f9767c-workload-socket" pod "2b281dea-bfcb-4edf-a421-d1bff5f9767c" (UID: "2b281dea-bfcb-4edf-a421-d1bff5f9767c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:48:59.128157 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:59.128135 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b281dea-bfcb-4edf-a421-d1bff5f9767c-workload-certs" pod "2b281dea-bfcb-4edf-a421-d1bff5f9767c" (UID: "2b281dea-bfcb-4edf-a421-d1bff5f9767c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:48:59.128364 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:59.128347 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b281dea-bfcb-4edf-a421-d1bff5f9767c-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:48:59.128441 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:59.128371 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b281dea-bfcb-4edf-a421-d1bff5f9767c-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:48:59.128441 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:59.128384 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b281dea-bfcb-4edf-a421-d1bff5f9767c-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:48:59.128441 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:59.128398 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b281dea-bfcb-4edf-a421-d1bff5f9767c-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:48:59.128441 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:59.128412 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b281dea-bfcb-4edf-a421-d1bff5f9767c-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:48:59.129546 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:59.129524 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b281dea-bfcb-4edf-a421-d1bff5f9767c-istio-podinfo" pod "2b281dea-bfcb-4edf-a421-d1bff5f9767c" (UID: "2b281dea-bfcb-4edf-a421-d1bff5f9767c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:48:59.129649 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:59.129635 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b281dea-bfcb-4edf-a421-d1bff5f9767c-istio-token" pod "2b281dea-bfcb-4edf-a421-d1bff5f9767c" (UID: "2b281dea-bfcb-4edf-a421-d1bff5f9767c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:48:59.129953 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:59.129934 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b281dea-bfcb-4edf-a421-d1bff5f9767c-kube-api-access-955hz" pod "2b281dea-bfcb-4edf-a421-d1bff5f9767c" (UID: "2b281dea-bfcb-4edf-a421-d1bff5f9767c"). InnerVolumeSpecName "kube-api-access-955hz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:48:59.130031 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:59.129988 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b281dea-bfcb-4edf-a421-d1bff5f9767c-istio-envoy" pod "2b281dea-bfcb-4edf-a421-d1bff5f9767c" (UID: "2b281dea-bfcb-4edf-a421-d1bff5f9767c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:48:59.230017 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:59.229982 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b281dea-bfcb-4edf-a421-d1bff5f9767c-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:48:59.230017 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:59.230010 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b281dea-bfcb-4edf-a421-d1bff5f9767c-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:48:59.230017 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:59.230020 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-955hz\" (UniqueName: \"kubernetes.io/projected/2b281dea-bfcb-4edf-a421-d1bff5f9767c-kube-api-access-955hz\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:48:59.230258 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:59.230030 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b281dea-bfcb-4edf-a421-d1bff5f9767c-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:48:59.620951 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:59.620916 5076 generic.go:356] "Generic (PLEG): container finished" podID="2b281dea-bfcb-4edf-a421-d1bff5f9767c" containerID="84e2e0a98dc6dd0db04cb123307f3707aba86be25c3629301de9b4328b09b7d4" exitCode=0 Nov 02 23:48:59.621148 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:59.620961 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" Nov 02 23:48:59.621148 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:59.620997 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" event={"ID":"2b281dea-bfcb-4edf-a421-d1bff5f9767c","Type":"ContainerDied","Data":"84e2e0a98dc6dd0db04cb123307f3707aba86be25c3629301de9b4328b09b7d4"} Nov 02 23:48:59.621148 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:59.621048 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5" event={"ID":"2b281dea-bfcb-4edf-a421-d1bff5f9767c","Type":"ContainerDied","Data":"9672127ab20da8a127f26315c619699c1f803ba9cfc97583ba868daca47473e9"} Nov 02 23:48:59.621148 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:59.621071 5076 scope.go:122] "RemoveContainer" containerID="84e2e0a98dc6dd0db04cb123307f3707aba86be25c3629301de9b4328b09b7d4" Nov 02 23:48:59.630732 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:59.630715 5076 scope.go:122] "RemoveContainer" containerID="84e2e0a98dc6dd0db04cb123307f3707aba86be25c3629301de9b4328b09b7d4" Nov 02 23:48:59.631055 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:48:59.631026 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84e2e0a98dc6dd0db04cb123307f3707aba86be25c3629301de9b4328b09b7d4\": container with ID starting with 84e2e0a98dc6dd0db04cb123307f3707aba86be25c3629301de9b4328b09b7d4 not found: ID does not exist" containerID="84e2e0a98dc6dd0db04cb123307f3707aba86be25c3629301de9b4328b09b7d4" Nov 02 23:48:59.631146 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:59.631055 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84e2e0a98dc6dd0db04cb123307f3707aba86be25c3629301de9b4328b09b7d4"} err="failed to get container status \"84e2e0a98dc6dd0db04cb123307f3707aba86be25c3629301de9b4328b09b7d4\": rpc error: code = NotFound desc = could not find container \"84e2e0a98dc6dd0db04cb123307f3707aba86be25c3629301de9b4328b09b7d4\": container with ID starting with 84e2e0a98dc6dd0db04cb123307f3707aba86be25c3629301de9b4328b09b7d4 not found: ID does not exist" Nov 02 23:48:59.778261 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:59.778228 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5"] Nov 02 23:48:59.781667 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:48:59.781643 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-313523-openshift-default-6fd45f8999-ncsh5"] Nov 02 23:49:01.149819 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:01.149784 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b281dea-bfcb-4edf-a421-d1bff5f9767c" path="/var/lib/kubelet/pods/2b281dea-bfcb-4edf-a421-d1bff5f9767c/volumes" Nov 02 23:49:09.921498 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:49:09.921448 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:49:10.711640 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:10.711605 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-fbec82/auth-test-model-fbec82-kserve-566f67c557-f9g2v"] Nov 02 23:49:23.517339 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.517304 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x"] Nov 02 23:49:23.517966 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.517949 5076 cpu_manager.go:414] "containerMap: removing container" podUID="2b281dea-bfcb-4edf-a421-d1bff5f9767c" containerName="istio-proxy" Nov 02 23:49:23.517966 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.517966 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b281dea-bfcb-4edf-a421-d1bff5f9767c" containerName="istio-proxy" Nov 02 23:49:23.518097 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.517987 5076 cpu_manager.go:414] "containerMap: removing container" podUID="ae04339e-7f59-4edd-862a-aa88108d59bb" containerName="istio-proxy" Nov 02 23:49:23.518097 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.517993 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae04339e-7f59-4edd-862a-aa88108d59bb" containerName="istio-proxy" Nov 02 23:49:23.518097 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.518015 5076 cpu_manager.go:414] "containerMap: removing container" podUID="ae04339e-7f59-4edd-862a-aa88108d59bb" containerName="istio-proxy" Nov 02 23:49:23.518097 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.518023 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae04339e-7f59-4edd-862a-aa88108d59bb" containerName="istio-proxy" Nov 02 23:49:23.518223 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.518116 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b281dea-bfcb-4edf-a421-d1bff5f9767c" containerName="istio-proxy" Nov 02 23:49:23.518223 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.518126 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fec430c-8002-4852-b0d2-32217cffb96b" containerName="cleanup" Nov 02 23:49:23.518223 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.518132 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae04339e-7f59-4edd-862a-aa88108d59bb" containerName="istio-proxy" Nov 02 23:49:23.518223 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.518141 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae04339e-7f59-4edd-862a-aa88108d59bb" containerName="istio-proxy" Nov 02 23:49:23.521533 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.521515 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" Nov 02 23:49:23.607951 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.602716 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x"] Nov 02 23:49:23.622950 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.622916 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-3bcb9f-openshift-default-dockercfg-br2qj\"" Nov 02 23:49:23.637133 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.627752 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2656fde-164f-40ec-845f-ad9c08091597-credential-socket\") pod \"e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x\" (UID: \"d2656fde-164f-40ec-845f-ad9c08091597\") " pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" Nov 02 23:49:23.637133 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.627838 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2656fde-164f-40ec-845f-ad9c08091597-workload-certs\") pod \"e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x\" (UID: \"d2656fde-164f-40ec-845f-ad9c08091597\") " pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" Nov 02 23:49:23.637133 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.627868 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2656fde-164f-40ec-845f-ad9c08091597-istio-data\") pod \"e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x\" (UID: \"d2656fde-164f-40ec-845f-ad9c08091597\") " pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" Nov 02 23:49:23.637592 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.637562 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2656fde-164f-40ec-845f-ad9c08091597-istiod-ca-cert\") pod \"e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x\" (UID: \"d2656fde-164f-40ec-845f-ad9c08091597\") " pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" Nov 02 23:49:23.637916 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.637880 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2656fde-164f-40ec-845f-ad9c08091597-workload-socket\") pod \"e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x\" (UID: \"d2656fde-164f-40ec-845f-ad9c08091597\") " pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" Nov 02 23:49:23.638056 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.638039 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4rnl\" (UniqueName: \"kubernetes.io/projected/d2656fde-164f-40ec-845f-ad9c08091597-kube-api-access-z4rnl\") pod \"e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x\" (UID: \"d2656fde-164f-40ec-845f-ad9c08091597\") " pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" Nov 02 23:49:23.638157 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.638145 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2656fde-164f-40ec-845f-ad9c08091597-istio-envoy\") pod \"e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x\" (UID: \"d2656fde-164f-40ec-845f-ad9c08091597\") " pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" Nov 02 23:49:23.638294 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.638282 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2656fde-164f-40ec-845f-ad9c08091597-istio-token\") pod \"e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x\" (UID: \"d2656fde-164f-40ec-845f-ad9c08091597\") " pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" Nov 02 23:49:23.638417 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.638405 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2656fde-164f-40ec-845f-ad9c08091597-istio-podinfo\") pod \"e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x\" (UID: \"d2656fde-164f-40ec-845f-ad9c08091597\") " pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" Nov 02 23:49:23.739244 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.739206 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2656fde-164f-40ec-845f-ad9c08091597-istio-token\") pod \"e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x\" (UID: \"d2656fde-164f-40ec-845f-ad9c08091597\") " pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" Nov 02 23:49:23.739420 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.739263 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2656fde-164f-40ec-845f-ad9c08091597-istio-podinfo\") pod \"e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x\" (UID: \"d2656fde-164f-40ec-845f-ad9c08091597\") " pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" Nov 02 23:49:23.739420 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.739341 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2656fde-164f-40ec-845f-ad9c08091597-credential-socket\") pod \"e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x\" (UID: \"d2656fde-164f-40ec-845f-ad9c08091597\") " pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" Nov 02 23:49:23.739420 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.739404 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2656fde-164f-40ec-845f-ad9c08091597-workload-certs\") pod \"e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x\" (UID: \"d2656fde-164f-40ec-845f-ad9c08091597\") " pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" Nov 02 23:49:23.739587 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.739430 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2656fde-164f-40ec-845f-ad9c08091597-istio-data\") pod \"e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x\" (UID: \"d2656fde-164f-40ec-845f-ad9c08091597\") " pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" Nov 02 23:49:23.739587 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.739564 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2656fde-164f-40ec-845f-ad9c08091597-istiod-ca-cert\") pod \"e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x\" (UID: \"d2656fde-164f-40ec-845f-ad9c08091597\") " pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" Nov 02 23:49:23.739680 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.739666 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2656fde-164f-40ec-845f-ad9c08091597-workload-socket\") pod \"e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x\" (UID: \"d2656fde-164f-40ec-845f-ad9c08091597\") " pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" Nov 02 23:49:23.739730 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.739698 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4rnl\" (UniqueName: \"kubernetes.io/projected/d2656fde-164f-40ec-845f-ad9c08091597-kube-api-access-z4rnl\") pod \"e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x\" (UID: \"d2656fde-164f-40ec-845f-ad9c08091597\") " pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" Nov 02 23:49:23.739791 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.739735 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2656fde-164f-40ec-845f-ad9c08091597-istio-envoy\") pod \"e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x\" (UID: \"d2656fde-164f-40ec-845f-ad9c08091597\") " pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" Nov 02 23:49:23.739943 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.739880 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2656fde-164f-40ec-845f-ad9c08091597-credential-socket\") pod \"e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x\" (UID: \"d2656fde-164f-40ec-845f-ad9c08091597\") " pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" Nov 02 23:49:23.739943 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.739928 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2656fde-164f-40ec-845f-ad9c08091597-istio-data\") pod \"e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x\" (UID: \"d2656fde-164f-40ec-845f-ad9c08091597\") " pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" Nov 02 23:49:23.740103 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.740022 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2656fde-164f-40ec-845f-ad9c08091597-workload-certs\") pod \"e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x\" (UID: \"d2656fde-164f-40ec-845f-ad9c08091597\") " pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" Nov 02 23:49:23.740301 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.740275 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2656fde-164f-40ec-845f-ad9c08091597-workload-socket\") pod \"e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x\" (UID: \"d2656fde-164f-40ec-845f-ad9c08091597\") " pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" Nov 02 23:49:23.740397 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.740378 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2656fde-164f-40ec-845f-ad9c08091597-istiod-ca-cert\") pod \"e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x\" (UID: \"d2656fde-164f-40ec-845f-ad9c08091597\") " pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" Nov 02 23:49:23.741916 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.741866 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2656fde-164f-40ec-845f-ad9c08091597-istio-envoy\") pod \"e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x\" (UID: \"d2656fde-164f-40ec-845f-ad9c08091597\") " pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" Nov 02 23:49:23.742095 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.742075 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2656fde-164f-40ec-845f-ad9c08091597-istio-podinfo\") pod \"e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x\" (UID: \"d2656fde-164f-40ec-845f-ad9c08091597\") " pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" Nov 02 23:49:23.793264 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.793194 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2656fde-164f-40ec-845f-ad9c08091597-istio-token\") pod \"e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x\" (UID: \"d2656fde-164f-40ec-845f-ad9c08091597\") " pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" Nov 02 23:49:23.793462 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.793358 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4rnl\" (UniqueName: \"kubernetes.io/projected/d2656fde-164f-40ec-845f-ad9c08091597-kube-api-access-z4rnl\") pod \"e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x\" (UID: \"d2656fde-164f-40ec-845f-ad9c08091597\") " pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" Nov 02 23:49:23.835998 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:23.835960 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" Nov 02 23:49:24.263712 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:24.263686 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x"] Nov 02 23:49:24.264337 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:49:24.264309 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2656fde_164f_40ec_845f_ad9c08091597.slice/crio-abf60cb25aae88dcc129f025ec8c810db2848e409d63a34c80b413ed80ed11a3 WatchSource:0}: Error finding container abf60cb25aae88dcc129f025ec8c810db2848e409d63a34c80b413ed80ed11a3: Status 404 returned error can't find the container with id abf60cb25aae88dcc129f025ec8c810db2848e409d63a34c80b413ed80ed11a3 Nov 02 23:49:24.266821 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:24.266786 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:49:24.266919 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:24.266855 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:49:24.266919 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:24.266884 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:49:24.772714 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:24.772674 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" event={"ID":"d2656fde-164f-40ec-845f-ad9c08091597","Type":"ContainerStarted","Data":"4761c7f31292c82638a9e7003c366e65f2dfa51e398e2b9631698ea65367f329"} Nov 02 23:49:24.773143 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:24.772736 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" event={"ID":"d2656fde-164f-40ec-845f-ad9c08091597","Type":"ContainerStarted","Data":"abf60cb25aae88dcc129f025ec8c810db2848e409d63a34c80b413ed80ed11a3"} Nov 02 23:49:24.836934 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:24.836885 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" Nov 02 23:49:24.884943 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:24.884852 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" podStartSLOduration=1.884836574 podStartE2EDuration="1.884836574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:49:24.88417603 +0000 UTC m=+3413.852707816" watchObservedRunningTime="2026-11-02 23:49:24.884836574 +0000 UTC m=+3413.853368357" Nov 02 23:49:25.732946 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:25.732880 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-fbec82/auth-test-model-fbec82-kserve-566f67c557-f9g2v" podUID="7037c73c-59cf-4f54-a054-201b076b62cb" containerName="main" containerID="cri-o://9cd4571cefabf36b85fae1db78f28cc9b46a92527e037607b5e19a1f733e35ed" gracePeriod=45 Nov 02 23:49:25.842925 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:25.842878 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" Nov 02 23:49:26.008710 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:26.008685 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-fbec82/auth-test-model-fbec82-kserve-566f67c557-f9g2v" Nov 02 23:49:26.065831 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:26.065798 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7037c73c-59cf-4f54-a054-201b076b62cb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7037c73c-59cf-4f54-a054-201b076b62cb-tmp-dir\") pod \"7037c73c-59cf-4f54-a054-201b076b62cb\" (UID: \"7037c73c-59cf-4f54-a054-201b076b62cb\") " Nov 02 23:49:26.066030 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:26.065992 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7037c73c-59cf-4f54-a054-201b076b62cb-kube-api-access-kzdm6\" (UniqueName: \"kubernetes.io/projected/7037c73c-59cf-4f54-a054-201b076b62cb-kube-api-access-kzdm6\") pod \"7037c73c-59cf-4f54-a054-201b076b62cb\" (UID: \"7037c73c-59cf-4f54-a054-201b076b62cb\") " Nov 02 23:49:26.066084 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:26.066026 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7037c73c-59cf-4f54-a054-201b076b62cb-home\" (UniqueName: \"kubernetes.io/empty-dir/7037c73c-59cf-4f54-a054-201b076b62cb-home\") pod \"7037c73c-59cf-4f54-a054-201b076b62cb\" (UID: \"7037c73c-59cf-4f54-a054-201b076b62cb\") " Nov 02 23:49:26.066084 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:26.066062 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7037c73c-59cf-4f54-a054-201b076b62cb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7037c73c-59cf-4f54-a054-201b076b62cb-dshm\") pod \"7037c73c-59cf-4f54-a054-201b076b62cb\" (UID: \"7037c73c-59cf-4f54-a054-201b076b62cb\") " Nov 02 23:49:26.066161 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:26.066091 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7037c73c-59cf-4f54-a054-201b076b62cb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7037c73c-59cf-4f54-a054-201b076b62cb-model-cache\") pod \"7037c73c-59cf-4f54-a054-201b076b62cb\" (UID: \"7037c73c-59cf-4f54-a054-201b076b62cb\") " Nov 02 23:49:26.066161 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:26.066133 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7037c73c-59cf-4f54-a054-201b076b62cb-tmp-dir" pod "7037c73c-59cf-4f54-a054-201b076b62cb" (UID: "7037c73c-59cf-4f54-a054-201b076b62cb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:49:26.066242 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:26.066167 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7037c73c-59cf-4f54-a054-201b076b62cb-tls-certs\" (UniqueName: \"kubernetes.io/secret/7037c73c-59cf-4f54-a054-201b076b62cb-tls-certs\") pod \"7037c73c-59cf-4f54-a054-201b076b62cb\" (UID: \"7037c73c-59cf-4f54-a054-201b076b62cb\") " Nov 02 23:49:26.066322 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:26.066297 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7037c73c-59cf-4f54-a054-201b076b62cb-home" pod "7037c73c-59cf-4f54-a054-201b076b62cb" (UID: "7037c73c-59cf-4f54-a054-201b076b62cb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:49:26.066375 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:26.066355 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7037c73c-59cf-4f54-a054-201b076b62cb-model-cache" pod "7037c73c-59cf-4f54-a054-201b076b62cb" (UID: "7037c73c-59cf-4f54-a054-201b076b62cb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:49:26.066627 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:26.066610 5076 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7037c73c-59cf-4f54-a054-201b076b62cb-home\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:49:26.066705 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:26.066628 5076 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7037c73c-59cf-4f54-a054-201b076b62cb-model-cache\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:49:26.066705 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:26.066638 5076 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7037c73c-59cf-4f54-a054-201b076b62cb-tmp-dir\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:49:26.068316 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:26.068284 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7037c73c-59cf-4f54-a054-201b076b62cb-dshm" pod "7037c73c-59cf-4f54-a054-201b076b62cb" (UID: "7037c73c-59cf-4f54-a054-201b076b62cb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:49:26.068432 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:26.068360 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7037c73c-59cf-4f54-a054-201b076b62cb-kube-api-access-kzdm6" pod "7037c73c-59cf-4f54-a054-201b076b62cb" (UID: "7037c73c-59cf-4f54-a054-201b076b62cb"). InnerVolumeSpecName "kube-api-access-kzdm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:49:26.068432 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:26.068383 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7037c73c-59cf-4f54-a054-201b076b62cb-tls-certs" pod "7037c73c-59cf-4f54-a054-201b076b62cb" (UID: "7037c73c-59cf-4f54-a054-201b076b62cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:49:26.167672 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:26.167629 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzdm6\" (UniqueName: \"kubernetes.io/projected/7037c73c-59cf-4f54-a054-201b076b62cb-kube-api-access-kzdm6\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:49:26.167672 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:26.167664 5076 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7037c73c-59cf-4f54-a054-201b076b62cb-dshm\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:49:26.167672 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:26.167677 5076 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7037c73c-59cf-4f54-a054-201b076b62cb-tls-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:49:26.792361 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:26.792326 5076 generic.go:356] "Generic (PLEG): container finished" podID="7037c73c-59cf-4f54-a054-201b076b62cb" containerID="9cd4571cefabf36b85fae1db78f28cc9b46a92527e037607b5e19a1f733e35ed" exitCode=0 Nov 02 23:49:26.792537 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:26.792389 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-fbec82/auth-test-model-fbec82-kserve-566f67c557-f9g2v" Nov 02 23:49:26.792537 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:26.792403 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-fbec82/auth-test-model-fbec82-kserve-566f67c557-f9g2v" event={"ID":"7037c73c-59cf-4f54-a054-201b076b62cb","Type":"ContainerDied","Data":"9cd4571cefabf36b85fae1db78f28cc9b46a92527e037607b5e19a1f733e35ed"} Nov 02 23:49:26.792537 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:26.792443 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-fbec82/auth-test-model-fbec82-kserve-566f67c557-f9g2v" event={"ID":"7037c73c-59cf-4f54-a054-201b076b62cb","Type":"ContainerDied","Data":"d61c09c69b10705932cd9a34a96bf04bf45121f12a29abd2827db70dff1287ae"} Nov 02 23:49:26.792537 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:26.792459 5076 scope.go:122] "RemoveContainer" containerID="9cd4571cefabf36b85fae1db78f28cc9b46a92527e037607b5e19a1f733e35ed" Nov 02 23:49:26.793161 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:26.793044 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" Nov 02 23:49:26.794233 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:26.794192 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" Nov 02 23:49:26.803446 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:26.803430 5076 scope.go:122] "RemoveContainer" containerID="9cd4571cefabf36b85fae1db78f28cc9b46a92527e037607b5e19a1f733e35ed" Nov 02 23:49:26.803709 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:49:26.803690 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cd4571cefabf36b85fae1db78f28cc9b46a92527e037607b5e19a1f733e35ed\": container with ID starting with 9cd4571cefabf36b85fae1db78f28cc9b46a92527e037607b5e19a1f733e35ed not found: ID does not exist" containerID="9cd4571cefabf36b85fae1db78f28cc9b46a92527e037607b5e19a1f733e35ed" Nov 02 23:49:26.803758 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:26.803717 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cd4571cefabf36b85fae1db78f28cc9b46a92527e037607b5e19a1f733e35ed"} err="failed to get container status \"9cd4571cefabf36b85fae1db78f28cc9b46a92527e037607b5e19a1f733e35ed\": rpc error: code = NotFound desc = could not find container \"9cd4571cefabf36b85fae1db78f28cc9b46a92527e037607b5e19a1f733e35ed\": container with ID starting with 9cd4571cefabf36b85fae1db78f28cc9b46a92527e037607b5e19a1f733e35ed not found: ID does not exist" Nov 02 23:49:27.062483 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:27.062396 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-fbec82/auth-test-model-fbec82-kserve-566f67c557-f9g2v"] Nov 02 23:49:27.070577 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:27.070543 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-fbec82/auth-test-model-fbec82-kserve-566f67c557-f9g2v"] Nov 02 23:49:27.148411 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:27.148378 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7037c73c-59cf-4f54-a054-201b076b62cb" path="/var/lib/kubelet/pods/7037c73c-59cf-4f54-a054-201b076b62cb/volumes" Nov 02 23:49:38.235949 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:38.235855 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n"] Nov 02 23:49:38.236557 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:38.236536 5076 cpu_manager.go:414] "containerMap: removing container" podUID="7037c73c-59cf-4f54-a054-201b076b62cb" containerName="main" Nov 02 23:49:38.236636 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:38.236558 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7037c73c-59cf-4f54-a054-201b076b62cb" containerName="main" Nov 02 23:49:38.236701 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:38.236691 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="7037c73c-59cf-4f54-a054-201b076b62cb" containerName="main" Nov 02 23:49:38.240018 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:38.240002 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" Nov 02 23:49:38.291286 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:38.291250 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-8c2bff-openshift-default-dockercfg-6l89j\"" Nov 02 23:49:38.338865 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:38.338821 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-istio-podinfo\") pod \"e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n\" (UID: \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\") " pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" Nov 02 23:49:38.339247 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:38.339222 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggl8t\" (UniqueName: \"kubernetes.io/projected/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-kube-api-access-ggl8t\") pod \"e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n\" (UID: \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\") " pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" Nov 02 23:49:38.344846 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:38.344805 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-workload-socket\") pod \"e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n\" (UID: \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\") " pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" Nov 02 23:49:38.345113 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:38.345089 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-credential-socket\") pod \"e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n\" (UID: \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\") " pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" Nov 02 23:49:38.345329 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:38.345313 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-istio-envoy\") pod \"e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n\" (UID: \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\") " pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" Nov 02 23:49:38.345454 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:38.345439 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-istio-data\") pod \"e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n\" (UID: \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\") " pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" Nov 02 23:49:38.346057 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:38.345631 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-istiod-ca-cert\") pod \"e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n\" (UID: \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\") " pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" Nov 02 23:49:38.346288 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:38.346263 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-workload-certs\") pod \"e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n\" (UID: \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\") " pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" Nov 02 23:49:38.346457 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:38.346441 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-istio-token\") pod \"e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n\" (UID: \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\") " pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" Nov 02 23:49:38.349203 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:38.349181 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n"] Nov 02 23:49:38.447478 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:38.447439 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-istio-envoy\") pod \"e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n\" (UID: \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\") " pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" Nov 02 23:49:38.447478 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:38.447481 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-istio-data\") pod \"e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n\" (UID: \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\") " pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" Nov 02 23:49:38.447728 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:38.447579 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-istiod-ca-cert\") pod \"e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n\" (UID: \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\") " pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" Nov 02 23:49:38.447728 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:38.447629 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-workload-certs\") pod \"e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n\" (UID: \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\") " pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" Nov 02 23:49:38.447728 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:38.447683 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-istio-token\") pod \"e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n\" (UID: \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\") " pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" Nov 02 23:49:38.447728 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:38.447713 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-istio-podinfo\") pod \"e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n\" (UID: \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\") " pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" Nov 02 23:49:38.447933 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:38.447781 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggl8t\" (UniqueName: \"kubernetes.io/projected/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-kube-api-access-ggl8t\") pod \"e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n\" (UID: \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\") " pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" Nov 02 23:49:38.447933 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:38.447809 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-workload-socket\") pod \"e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n\" (UID: \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\") " pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" Nov 02 23:49:38.447933 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:38.447847 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-credential-socket\") pod \"e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n\" (UID: \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\") " pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" Nov 02 23:49:38.448101 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:38.448081 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-workload-certs\") pod \"e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n\" (UID: \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\") " pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" Nov 02 23:49:38.448248 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:38.448224 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-credential-socket\") pod \"e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n\" (UID: \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\") " pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" Nov 02 23:49:38.448331 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:38.448279 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-workload-socket\") pod \"e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n\" (UID: \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\") " pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" Nov 02 23:49:38.448393 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:38.448360 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-istio-data\") pod \"e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n\" (UID: \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\") " pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" Nov 02 23:49:38.448432 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:38.448409 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-istiod-ca-cert\") pod \"e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n\" (UID: \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\") " pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" Nov 02 23:49:38.450131 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:38.450103 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-istio-envoy\") pod \"e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n\" (UID: \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\") " pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" Nov 02 23:49:38.450498 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:38.450479 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-istio-podinfo\") pod \"e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n\" (UID: \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\") " pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" Nov 02 23:49:38.514253 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:38.514175 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-istio-token\") pod \"e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n\" (UID: \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\") " pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" Nov 02 23:49:38.514415 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:38.514362 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggl8t\" (UniqueName: \"kubernetes.io/projected/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-kube-api-access-ggl8t\") pod \"e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n\" (UID: \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\") " pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" Nov 02 23:49:38.568060 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:38.568026 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" Nov 02 23:49:38.987836 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:38.987798 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n"] Nov 02 23:49:39.023576 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:49:39.023542 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7947bd91_9f2d_44dc_9e12_d3c8f3744ab0.slice/crio-1967e77cd5fb50f35607ce483c912a927775a7edf98b629e1327fa37c81811e4 WatchSource:0}: Error finding container 1967e77cd5fb50f35607ce483c912a927775a7edf98b629e1327fa37c81811e4: Status 404 returned error can't find the container with id 1967e77cd5fb50f35607ce483c912a927775a7edf98b629e1327fa37c81811e4 Nov 02 23:49:39.026726 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:39.026684 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:49:39.026816 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:39.026780 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:49:39.026861 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:39.026830 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:49:39.880874 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:39.880838 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" event={"ID":"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0","Type":"ContainerStarted","Data":"e7a104e2d6299ccea44e62b5db27a57544f56dc0d2828eb158201b72a8c2a40d"} Nov 02 23:49:39.881295 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:39.880915 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" event={"ID":"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0","Type":"ContainerStarted","Data":"1967e77cd5fb50f35607ce483c912a927775a7edf98b629e1327fa37c81811e4"} Nov 02 23:49:39.885282 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:49:39.885251 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:49:39.994214 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:39.994156 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" podStartSLOduration=1.994141034 podStartE2EDuration="1.994141034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:49:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:49:39.992217438 +0000 UTC m=+3428.960749225" watchObservedRunningTime="2026-11-02 23:49:39.994141034 +0000 UTC m=+3428.962672902" Nov 02 23:49:40.568232 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:40.568198 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" Nov 02 23:49:40.573765 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:40.573739 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" Nov 02 23:49:40.887149 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:40.887072 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" Nov 02 23:49:40.887942 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:49:40.887920 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" Nov 02 23:50:09.884954 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:50:09.884910 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:50:24.819837 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:24.819797 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6"] Nov 02 23:50:24.824183 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:24.824160 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" Nov 02 23:50:24.886574 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:24.886533 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-e65845-openshift-default-dockercfg-fgqjd\"" Nov 02 23:50:24.928061 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:24.928028 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6"] Nov 02 23:50:24.967575 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:24.967534 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m98cd\" (UniqueName: \"kubernetes.io/projected/000f0ea7-6931-4670-9d4f-814c072405c9-kube-api-access-m98cd\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6\" (UID: \"000f0ea7-6931-4670-9d4f-814c072405c9\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" Nov 02 23:50:24.967777 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:24.967587 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/000f0ea7-6931-4670-9d4f-814c072405c9-istio-token\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6\" (UID: \"000f0ea7-6931-4670-9d4f-814c072405c9\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" Nov 02 23:50:24.967777 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:24.967738 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/000f0ea7-6931-4670-9d4f-814c072405c9-workload-socket\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6\" (UID: \"000f0ea7-6931-4670-9d4f-814c072405c9\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" Nov 02 23:50:24.967777 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:24.967763 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/000f0ea7-6931-4670-9d4f-814c072405c9-istio-envoy\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6\" (UID: \"000f0ea7-6931-4670-9d4f-814c072405c9\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" Nov 02 23:50:24.967980 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:24.967805 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/000f0ea7-6931-4670-9d4f-814c072405c9-istiod-ca-cert\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6\" (UID: \"000f0ea7-6931-4670-9d4f-814c072405c9\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" Nov 02 23:50:24.967980 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:24.967863 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/000f0ea7-6931-4670-9d4f-814c072405c9-istio-podinfo\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6\" (UID: \"000f0ea7-6931-4670-9d4f-814c072405c9\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" Nov 02 23:50:24.967980 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:24.967924 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/000f0ea7-6931-4670-9d4f-814c072405c9-workload-certs\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6\" (UID: \"000f0ea7-6931-4670-9d4f-814c072405c9\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" Nov 02 23:50:24.967980 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:24.967973 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/000f0ea7-6931-4670-9d4f-814c072405c9-credential-socket\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6\" (UID: \"000f0ea7-6931-4670-9d4f-814c072405c9\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" Nov 02 23:50:24.968239 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:24.968019 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/000f0ea7-6931-4670-9d4f-814c072405c9-istio-data\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6\" (UID: \"000f0ea7-6931-4670-9d4f-814c072405c9\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" Nov 02 23:50:25.072653 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:25.070039 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/000f0ea7-6931-4670-9d4f-814c072405c9-workload-socket\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6\" (UID: \"000f0ea7-6931-4670-9d4f-814c072405c9\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" Nov 02 23:50:25.072653 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:25.070092 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/000f0ea7-6931-4670-9d4f-814c072405c9-istio-envoy\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6\" (UID: \"000f0ea7-6931-4670-9d4f-814c072405c9\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" Nov 02 23:50:25.072653 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:25.070140 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/000f0ea7-6931-4670-9d4f-814c072405c9-istiod-ca-cert\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6\" (UID: \"000f0ea7-6931-4670-9d4f-814c072405c9\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" Nov 02 23:50:25.072653 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:25.070205 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/000f0ea7-6931-4670-9d4f-814c072405c9-istio-podinfo\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6\" (UID: \"000f0ea7-6931-4670-9d4f-814c072405c9\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" Nov 02 23:50:25.072653 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:25.070254 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/000f0ea7-6931-4670-9d4f-814c072405c9-workload-certs\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6\" (UID: \"000f0ea7-6931-4670-9d4f-814c072405c9\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" Nov 02 23:50:25.072653 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:25.070310 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/000f0ea7-6931-4670-9d4f-814c072405c9-credential-socket\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6\" (UID: \"000f0ea7-6931-4670-9d4f-814c072405c9\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" Nov 02 23:50:25.072653 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:25.070377 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/000f0ea7-6931-4670-9d4f-814c072405c9-istio-data\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6\" (UID: \"000f0ea7-6931-4670-9d4f-814c072405c9\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" Nov 02 23:50:25.072653 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:25.070436 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m98cd\" (UniqueName: \"kubernetes.io/projected/000f0ea7-6931-4670-9d4f-814c072405c9-kube-api-access-m98cd\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6\" (UID: \"000f0ea7-6931-4670-9d4f-814c072405c9\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" Nov 02 23:50:25.072653 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:25.070462 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/000f0ea7-6931-4670-9d4f-814c072405c9-istio-token\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6\" (UID: \"000f0ea7-6931-4670-9d4f-814c072405c9\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" Nov 02 23:50:25.072653 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:25.071294 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/000f0ea7-6931-4670-9d4f-814c072405c9-workload-socket\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6\" (UID: \"000f0ea7-6931-4670-9d4f-814c072405c9\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" Nov 02 23:50:25.072653 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:25.071914 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/000f0ea7-6931-4670-9d4f-814c072405c9-workload-certs\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6\" (UID: \"000f0ea7-6931-4670-9d4f-814c072405c9\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" Nov 02 23:50:25.072653 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:25.072591 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/000f0ea7-6931-4670-9d4f-814c072405c9-istiod-ca-cert\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6\" (UID: \"000f0ea7-6931-4670-9d4f-814c072405c9\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" Nov 02 23:50:25.073401 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:25.072840 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/000f0ea7-6931-4670-9d4f-814c072405c9-istio-data\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6\" (UID: \"000f0ea7-6931-4670-9d4f-814c072405c9\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" Nov 02 23:50:25.073401 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:25.073110 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/000f0ea7-6931-4670-9d4f-814c072405c9-credential-socket\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6\" (UID: \"000f0ea7-6931-4670-9d4f-814c072405c9\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" Nov 02 23:50:25.076977 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:25.074738 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/000f0ea7-6931-4670-9d4f-814c072405c9-istio-envoy\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6\" (UID: \"000f0ea7-6931-4670-9d4f-814c072405c9\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" Nov 02 23:50:25.078866 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:25.078829 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/000f0ea7-6931-4670-9d4f-814c072405c9-istio-podinfo\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6\" (UID: \"000f0ea7-6931-4670-9d4f-814c072405c9\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" Nov 02 23:50:25.129912 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:25.127905 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/000f0ea7-6931-4670-9d4f-814c072405c9-istio-token\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6\" (UID: \"000f0ea7-6931-4670-9d4f-814c072405c9\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" Nov 02 23:50:25.151924 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:25.130747 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m98cd\" (UniqueName: \"kubernetes.io/projected/000f0ea7-6931-4670-9d4f-814c072405c9-kube-api-access-m98cd\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6\" (UID: \"000f0ea7-6931-4670-9d4f-814c072405c9\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" Nov 02 23:50:25.151924 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:25.139388 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" Nov 02 23:50:25.636414 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:25.636378 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6"] Nov 02 23:50:25.637253 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:50:25.637217 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod000f0ea7_6931_4670_9d4f_814c072405c9.slice/crio-02a24b2390a7215e8767f2fecd5839e4a85bce93d10d94303a84d31a75f93b98 WatchSource:0}: Error finding container 02a24b2390a7215e8767f2fecd5839e4a85bce93d10d94303a84d31a75f93b98: Status 404 returned error can't find the container with id 02a24b2390a7215e8767f2fecd5839e4a85bce93d10d94303a84d31a75f93b98 Nov 02 23:50:25.639822 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:25.639785 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:50:25.639952 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:25.639852 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:50:25.639952 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:25.639882 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:50:26.211443 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:26.211406 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" event={"ID":"000f0ea7-6931-4670-9d4f-814c072405c9","Type":"ContainerStarted","Data":"7778c67442e41bf95c06a3aa3e62061243a047374935a420da52367cb9445e6c"} Nov 02 23:50:26.211998 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:26.211456 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" event={"ID":"000f0ea7-6931-4670-9d4f-814c072405c9","Type":"ContainerStarted","Data":"02a24b2390a7215e8767f2fecd5839e4a85bce93d10d94303a84d31a75f93b98"} Nov 02 23:50:26.323737 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:26.323685 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" podStartSLOduration=2.3236701269999998 podStartE2EDuration="2.323670127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:50:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:50:26.322086601 +0000 UTC m=+3475.290618387" watchObservedRunningTime="2026-11-02 23:50:26.323670127 +0000 UTC m=+3475.292201911" Nov 02 23:50:27.139767 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:27.139731 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" Nov 02 23:50:27.160798 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:27.160764 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" Nov 02 23:50:27.218202 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:27.218169 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" Nov 02 23:50:27.219081 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:27.219058 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" Nov 02 23:50:28.664349 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:28.664320 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-b4d53e/auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw"] Nov 02 23:50:34.049915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:34.048453 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw"] Nov 02 23:50:34.067392 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:34.066649 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw" Nov 02 23:50:34.099410 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:34.099378 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw"] Nov 02 23:50:34.143363 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:34.143326 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-3bcb9f-serving-cert\"" Nov 02 23:50:34.194652 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:34.194617 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/519907c3-2bc2-490b-a888-ee3b91ba5f8a-metrics-certs\") pod \"maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw\" (UID: \"519907c3-2bc2-490b-a888-ee3b91ba5f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw" Nov 02 23:50:34.194839 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:34.194752 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8pbx\" (UniqueName: \"kubernetes.io/projected/519907c3-2bc2-490b-a888-ee3b91ba5f8a-kube-api-access-x8pbx\") pod \"maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw\" (UID: \"519907c3-2bc2-490b-a888-ee3b91ba5f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw" Nov 02 23:50:34.194957 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:34.194938 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519907c3-2bc2-490b-a888-ee3b91ba5f8a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw\" (UID: \"519907c3-2bc2-490b-a888-ee3b91ba5f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw" Nov 02 23:50:34.195000 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:34.194970 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519907c3-2bc2-490b-a888-ee3b91ba5f8a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw\" (UID: \"519907c3-2bc2-490b-a888-ee3b91ba5f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw" Nov 02 23:50:34.336962 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:34.336094 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/519907c3-2bc2-490b-a888-ee3b91ba5f8a-metrics-certs\") pod \"maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw\" (UID: \"519907c3-2bc2-490b-a888-ee3b91ba5f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw" Nov 02 23:50:34.336962 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:34.336217 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8pbx\" (UniqueName: \"kubernetes.io/projected/519907c3-2bc2-490b-a888-ee3b91ba5f8a-kube-api-access-x8pbx\") pod \"maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw\" (UID: \"519907c3-2bc2-490b-a888-ee3b91ba5f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw" Nov 02 23:50:34.336962 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:34.336403 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519907c3-2bc2-490b-a888-ee3b91ba5f8a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw\" (UID: \"519907c3-2bc2-490b-a888-ee3b91ba5f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw" Nov 02 23:50:34.336962 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:34.336434 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519907c3-2bc2-490b-a888-ee3b91ba5f8a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw\" (UID: \"519907c3-2bc2-490b-a888-ee3b91ba5f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw" Nov 02 23:50:34.348946 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:34.337380 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519907c3-2bc2-490b-a888-ee3b91ba5f8a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw\" (UID: \"519907c3-2bc2-490b-a888-ee3b91ba5f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw" Nov 02 23:50:34.348946 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:50:34.338170 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-3bcb9f-serving-cert: secret "maas-api-e2e-extmod-a-3bcb9f-serving-cert" not found Nov 02 23:50:34.348946 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:50:34.338236 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/519907c3-2bc2-490b-a888-ee3b91ba5f8a-maas-api-tls podName:519907c3-2bc2-490b-a888-ee3b91ba5f8a nodeName:}" failed. No retries permitted until 2026-11-02 23:50:34.838215617 +0000 UTC m=+3483.806747397 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/519907c3-2bc2-490b-a888-ee3b91ba5f8a-maas-api-tls") pod "maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw" (UID: "519907c3-2bc2-490b-a888-ee3b91ba5f8a") : secret "maas-api-e2e-extmod-a-3bcb9f-serving-cert" not found Nov 02 23:50:34.361915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:34.357498 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/519907c3-2bc2-490b-a888-ee3b91ba5f8a-metrics-certs\") pod \"maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw\" (UID: \"519907c3-2bc2-490b-a888-ee3b91ba5f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw" Nov 02 23:50:34.410501 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:34.410466 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8pbx\" (UniqueName: \"kubernetes.io/projected/519907c3-2bc2-490b-a888-ee3b91ba5f8a-kube-api-access-x8pbx\") pod \"maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw\" (UID: \"519907c3-2bc2-490b-a888-ee3b91ba5f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw" Nov 02 23:50:34.843789 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:34.843754 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519907c3-2bc2-490b-a888-ee3b91ba5f8a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw\" (UID: \"519907c3-2bc2-490b-a888-ee3b91ba5f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw" Nov 02 23:50:34.846328 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:34.846304 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519907c3-2bc2-490b-a888-ee3b91ba5f8a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw\" (UID: \"519907c3-2bc2-490b-a888-ee3b91ba5f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw" Nov 02 23:50:34.995875 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:34.995838 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw" Nov 02 23:50:35.364007 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:35.363974 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw"] Nov 02 23:50:35.365381 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:50:35.365347 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod519907c3_2bc2_490b_a888_ee3b91ba5f8a.slice/crio-24ed1732fb0cdf265c76371dd563100ab4fbc747dedc4a025b7fec0acffebdd8 WatchSource:0}: Error finding container 24ed1732fb0cdf265c76371dd563100ab4fbc747dedc4a025b7fec0acffebdd8: Status 404 returned error can't find the container with id 24ed1732fb0cdf265c76371dd563100ab4fbc747dedc4a025b7fec0acffebdd8 Nov 02 23:50:35.367386 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:35.367363 5076 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 23:50:36.275822 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:36.275781 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw" event={"ID":"519907c3-2bc2-490b-a888-ee3b91ba5f8a","Type":"ContainerStarted","Data":"24ed1732fb0cdf265c76371dd563100ab4fbc747dedc4a025b7fec0acffebdd8"} Nov 02 23:50:37.283906 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:37.283855 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw" event={"ID":"519907c3-2bc2-490b-a888-ee3b91ba5f8a","Type":"ContainerStarted","Data":"ccdb4f79760af50d6de0c853c30256f0bc308f8741cc210b068890f8d7b5ccc0"} Nov 02 23:50:37.284421 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:37.283941 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw" Nov 02 23:50:37.396283 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:37.396230 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw" podStartSLOduration=3.423245825 podStartE2EDuration="4.396216588s" totalImagesPullingTime="972.970763ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:50:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:50:37.393832605 +0000 UTC m=+3486.362364391" watchObservedRunningTime="2026-11-02 23:50:37.396216588 +0000 UTC m=+3486.364748372" Nov 02 23:50:39.884591 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:50:39.884550 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:50:43.296862 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:43.296829 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw" Nov 02 23:50:43.684827 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:43.684787 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-b4d53e/auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw" podUID="8c816427-1b10-4355-b096-264ce632a2aa" containerName="main" containerID="cri-o://6f8dc9a364ae06e9fe04218d7b2d330dd5105edd5114d4e32bf76f5c14c07baf" gracePeriod=45 Nov 02 23:50:43.965573 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:43.965549 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-b4d53e/auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw" Nov 02 23:50:44.069030 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:44.068999 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c816427-1b10-4355-b096-264ce632a2aa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c816427-1b10-4355-b096-264ce632a2aa-tmp-dir\") pod \"8c816427-1b10-4355-b096-264ce632a2aa\" (UID: \"8c816427-1b10-4355-b096-264ce632a2aa\") " Nov 02 23:50:44.069213 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:44.069048 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c816427-1b10-4355-b096-264ce632a2aa-tls-certs\" (UniqueName: \"kubernetes.io/secret/8c816427-1b10-4355-b096-264ce632a2aa-tls-certs\") pod \"8c816427-1b10-4355-b096-264ce632a2aa\" (UID: \"8c816427-1b10-4355-b096-264ce632a2aa\") " Nov 02 23:50:44.069213 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:44.069149 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c816427-1b10-4355-b096-264ce632a2aa-kube-api-access-bzdzp\" (UniqueName: \"kubernetes.io/projected/8c816427-1b10-4355-b096-264ce632a2aa-kube-api-access-bzdzp\") pod \"8c816427-1b10-4355-b096-264ce632a2aa\" (UID: \"8c816427-1b10-4355-b096-264ce632a2aa\") " Nov 02 23:50:44.069285 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:44.069241 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c816427-1b10-4355-b096-264ce632a2aa-home\" (UniqueName: \"kubernetes.io/empty-dir/8c816427-1b10-4355-b096-264ce632a2aa-home\") pod \"8c816427-1b10-4355-b096-264ce632a2aa\" (UID: \"8c816427-1b10-4355-b096-264ce632a2aa\") " Nov 02 23:50:44.069285 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:44.069269 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c816427-1b10-4355-b096-264ce632a2aa-tmp-dir" pod "8c816427-1b10-4355-b096-264ce632a2aa" (UID: "8c816427-1b10-4355-b096-264ce632a2aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:50:44.069402 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:44.069318 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c816427-1b10-4355-b096-264ce632a2aa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c816427-1b10-4355-b096-264ce632a2aa-model-cache\") pod \"8c816427-1b10-4355-b096-264ce632a2aa\" (UID: \"8c816427-1b10-4355-b096-264ce632a2aa\") " Nov 02 23:50:44.069402 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:44.069348 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c816427-1b10-4355-b096-264ce632a2aa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c816427-1b10-4355-b096-264ce632a2aa-dshm\") pod \"8c816427-1b10-4355-b096-264ce632a2aa\" (UID: \"8c816427-1b10-4355-b096-264ce632a2aa\") " Nov 02 23:50:44.069548 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:44.069518 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c816427-1b10-4355-b096-264ce632a2aa-home" pod "8c816427-1b10-4355-b096-264ce632a2aa" (UID: "8c816427-1b10-4355-b096-264ce632a2aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:50:44.069676 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:44.069550 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c816427-1b10-4355-b096-264ce632a2aa-model-cache" pod "8c816427-1b10-4355-b096-264ce632a2aa" (UID: "8c816427-1b10-4355-b096-264ce632a2aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:50:44.069821 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:44.069803 5076 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c816427-1b10-4355-b096-264ce632a2aa-home\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:50:44.069880 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:44.069827 5076 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c816427-1b10-4355-b096-264ce632a2aa-model-cache\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:50:44.069880 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:44.069842 5076 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c816427-1b10-4355-b096-264ce632a2aa-tmp-dir\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:50:44.071517 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:44.071498 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c816427-1b10-4355-b096-264ce632a2aa-dshm" pod "8c816427-1b10-4355-b096-264ce632a2aa" (UID: "8c816427-1b10-4355-b096-264ce632a2aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:50:44.071803 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:44.071782 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c816427-1b10-4355-b096-264ce632a2aa-kube-api-access-bzdzp" pod "8c816427-1b10-4355-b096-264ce632a2aa" (UID: "8c816427-1b10-4355-b096-264ce632a2aa"). InnerVolumeSpecName "kube-api-access-bzdzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:50:44.071866 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:44.071781 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c816427-1b10-4355-b096-264ce632a2aa-tls-certs" pod "8c816427-1b10-4355-b096-264ce632a2aa" (UID: "8c816427-1b10-4355-b096-264ce632a2aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:50:44.170904 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:44.170865 5076 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c816427-1b10-4355-b096-264ce632a2aa-tls-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:50:44.171093 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:44.170913 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzdzp\" (UniqueName: \"kubernetes.io/projected/8c816427-1b10-4355-b096-264ce632a2aa-kube-api-access-bzdzp\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:50:44.171093 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:44.170931 5076 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c816427-1b10-4355-b096-264ce632a2aa-dshm\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:50:44.327809 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:44.327718 5076 generic.go:356] "Generic (PLEG): container finished" podID="8c816427-1b10-4355-b096-264ce632a2aa" containerID="6f8dc9a364ae06e9fe04218d7b2d330dd5105edd5114d4e32bf76f5c14c07baf" exitCode=0 Nov 02 23:50:44.327809 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:44.327765 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-b4d53e/auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw" Nov 02 23:50:44.328319 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:44.327804 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-b4d53e/auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw" event={"ID":"8c816427-1b10-4355-b096-264ce632a2aa","Type":"ContainerDied","Data":"6f8dc9a364ae06e9fe04218d7b2d330dd5105edd5114d4e32bf76f5c14c07baf"} Nov 02 23:50:44.328319 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:44.327850 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-b4d53e/auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw" event={"ID":"8c816427-1b10-4355-b096-264ce632a2aa","Type":"ContainerDied","Data":"6897c0a54d0a5382108bfc0153e4d8c24346a786c9e903f5fcf49db2c27ef76e"} Nov 02 23:50:44.328319 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:44.327873 5076 scope.go:122] "RemoveContainer" containerID="6f8dc9a364ae06e9fe04218d7b2d330dd5105edd5114d4e32bf76f5c14c07baf" Nov 02 23:50:44.338228 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:44.338210 5076 scope.go:122] "RemoveContainer" containerID="6f8dc9a364ae06e9fe04218d7b2d330dd5105edd5114d4e32bf76f5c14c07baf" Nov 02 23:50:44.338521 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:50:44.338502 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f8dc9a364ae06e9fe04218d7b2d330dd5105edd5114d4e32bf76f5c14c07baf\": container with ID starting with 6f8dc9a364ae06e9fe04218d7b2d330dd5105edd5114d4e32bf76f5c14c07baf not found: ID does not exist" containerID="6f8dc9a364ae06e9fe04218d7b2d330dd5105edd5114d4e32bf76f5c14c07baf" Nov 02 23:50:44.338575 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:44.338528 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f8dc9a364ae06e9fe04218d7b2d330dd5105edd5114d4e32bf76f5c14c07baf"} err="failed to get container status \"6f8dc9a364ae06e9fe04218d7b2d330dd5105edd5114d4e32bf76f5c14c07baf\": rpc error: code = NotFound desc = could not find container \"6f8dc9a364ae06e9fe04218d7b2d330dd5105edd5114d4e32bf76f5c14c07baf\": container with ID starting with 6f8dc9a364ae06e9fe04218d7b2d330dd5105edd5114d4e32bf76f5c14c07baf not found: ID does not exist" Nov 02 23:50:44.484938 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:44.484875 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-b4d53e/auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw"] Nov 02 23:50:44.487625 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:44.487603 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-b4d53e/auth-test-model-b4d53e-kserve-6744b5b8cc-qhptw"] Nov 02 23:50:45.147903 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:45.147862 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c816427-1b10-4355-b096-264ce632a2aa" path="/var/lib/kubelet/pods/8c816427-1b10-4355-b096-264ce632a2aa/volumes" Nov 02 23:50:52.378174 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:52.378140 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t"] Nov 02 23:50:52.378714 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:52.378700 5076 cpu_manager.go:414] "containerMap: removing container" podUID="8c816427-1b10-4355-b096-264ce632a2aa" containerName="main" Nov 02 23:50:52.378714 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:52.378715 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c816427-1b10-4355-b096-264ce632a2aa" containerName="main" Nov 02 23:50:52.378838 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:52.378829 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c816427-1b10-4355-b096-264ce632a2aa" containerName="main" Nov 02 23:50:52.383388 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:52.383370 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t" Nov 02 23:50:52.438370 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:52.438327 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t"] Nov 02 23:50:52.479733 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:52.479699 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-8c2bff-serving-cert\"" Nov 02 23:50:52.541627 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:52.534545 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t\" (UID: \"c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t" Nov 02 23:50:52.541627 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:52.541176 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t\" (UID: \"c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t" Nov 02 23:50:52.541627 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:52.541239 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbkwk\" (UniqueName: \"kubernetes.io/projected/c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa-kube-api-access-fbkwk\") pod \"maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t\" (UID: \"c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t" Nov 02 23:50:52.541627 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:52.541453 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa-metrics-certs\") pod \"maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t\" (UID: \"c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t" Nov 02 23:50:52.642882 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:52.642737 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa-metrics-certs\") pod \"maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t\" (UID: \"c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t" Nov 02 23:50:52.645016 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:52.644981 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t\" (UID: \"c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t" Nov 02 23:50:52.645328 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:52.645308 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t\" (UID: \"c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t" Nov 02 23:50:52.645471 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:52.645455 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbkwk\" (UniqueName: \"kubernetes.io/projected/c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa-kube-api-access-fbkwk\") pod \"maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t\" (UID: \"c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t" Nov 02 23:50:52.646860 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:52.646834 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t\" (UID: \"c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t" Nov 02 23:50:52.647129 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:50:52.647110 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c2bff-serving-cert: secret "maas-api-e2e-extmod-a-8c2bff-serving-cert" not found Nov 02 23:50:52.647322 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:50:52.647309 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa-maas-api-tls podName:c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa nodeName:}" failed. No retries permitted until 2026-11-02 23:50:53.147289357 +0000 UTC m=+3502.115821131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa-maas-api-tls") pod "maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t" (UID: "c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa") : secret "maas-api-e2e-extmod-a-8c2bff-serving-cert" not found Nov 02 23:50:52.664038 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:52.664002 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa-metrics-certs\") pod \"maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t\" (UID: \"c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t" Nov 02 23:50:52.699901 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:52.699857 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbkwk\" (UniqueName: \"kubernetes.io/projected/c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa-kube-api-access-fbkwk\") pod \"maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t\" (UID: \"c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t" Nov 02 23:50:53.152831 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:53.152797 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t\" (UID: \"c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t" Nov 02 23:50:53.155222 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:53.155200 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t\" (UID: \"c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t" Nov 02 23:50:53.296627 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:53.296589 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t" Nov 02 23:50:53.692661 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:53.692629 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t"] Nov 02 23:50:53.693416 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:50:53.693384 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5e0e172_c639_4b9a_b2ad_fb3cb8f837aa.slice/crio-88f0ad75e4f6a9f295b1a376aed6021866d78d5b8bd5a09dee46d0f0db4b1f9c WatchSource:0}: Error finding container 88f0ad75e4f6a9f295b1a376aed6021866d78d5b8bd5a09dee46d0f0db4b1f9c: Status 404 returned error can't find the container with id 88f0ad75e4f6a9f295b1a376aed6021866d78d5b8bd5a09dee46d0f0db4b1f9c Nov 02 23:50:54.439955 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:54.439916 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t" event={"ID":"c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa","Type":"ContainerStarted","Data":"88f0ad75e4f6a9f295b1a376aed6021866d78d5b8bd5a09dee46d0f0db4b1f9c"} Nov 02 23:50:55.450786 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:55.450745 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t" event={"ID":"c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa","Type":"ContainerStarted","Data":"0e3d3fb469e8fa4e94859c2802244196ae83eb92249761c7b9903881cd6d00a4"} Nov 02 23:50:55.451245 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:55.450882 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t" Nov 02 23:50:55.562373 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:55.562315 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t" podStartSLOduration=2.670171459 podStartE2EDuration="3.562300313s" totalImagesPullingTime="892.128854ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:50:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:50:55.559568403 +0000 UTC m=+3504.528100187" watchObservedRunningTime="2026-11-02 23:50:55.562300313 +0000 UTC m=+3504.530832097" Nov 02 23:50:57.834036 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:57.833996 5076 patch_prober.go:29] interesting pod/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:50:57.834536 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:57.834051 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" podUID="d2656fde-164f-40ec-845f-ad9c08091597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:50:59.476602 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:59.476559 5076 generic.go:356] "Generic (PLEG): container finished" podID="d2656fde-164f-40ec-845f-ad9c08091597" containerID="4761c7f31292c82638a9e7003c366e65f2dfa51e398e2b9631698ea65367f329" exitCode=0 Nov 02 23:50:59.477182 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:59.476637 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" event={"ID":"d2656fde-164f-40ec-845f-ad9c08091597","Type":"ContainerDied","Data":"4761c7f31292c82638a9e7003c366e65f2dfa51e398e2b9631698ea65367f329"} Nov 02 23:50:59.477182 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:59.477156 5076 scope.go:122] "RemoveContainer" containerID="4761c7f31292c82638a9e7003c366e65f2dfa51e398e2b9631698ea65367f329" Nov 02 23:50:59.478853 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:59.478819 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:50:59.478989 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:59.478919 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:50:59.478989 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:50:59.478953 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:51:00.485054 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:00.485019 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" event={"ID":"d2656fde-164f-40ec-845f-ad9c08091597","Type":"ContainerStarted","Data":"3cd4f87e29594cb2d3c3c442e9b8b343254094128c5552416704cee25dc14979"} Nov 02 23:51:00.837136 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:00.837027 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" Nov 02 23:51:00.837420 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:00.837251 5076 patch_prober.go:29] interesting pod/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 02 23:51:00.837568 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:00.837451 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" podUID="d2656fde-164f-40ec-845f-ad9c08091597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 02 23:51:01.468850 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:01.468820 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t" Nov 02 23:51:01.836710 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:01.836611 5076 patch_prober.go:29] interesting pod/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 02 23:51:01.836710 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:01.836690 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" podUID="d2656fde-164f-40ec-845f-ad9c08091597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 02 23:51:02.837139 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:02.837103 5076 patch_prober.go:29] interesting pod/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 02 23:51:02.837646 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:02.837164 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" podUID="d2656fde-164f-40ec-845f-ad9c08091597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 02 23:51:03.836488 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:03.836454 5076 patch_prober.go:29] interesting pod/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 02 23:51:03.836683 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:03.836524 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" podUID="d2656fde-164f-40ec-845f-ad9c08091597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 02 23:51:04.836982 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:04.836938 5076 patch_prober.go:29] interesting pod/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 02 23:51:04.837426 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:04.836998 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" podUID="d2656fde-164f-40ec-845f-ad9c08091597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 02 23:51:05.836306 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:05.836269 5076 patch_prober.go:29] interesting pod/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 02 23:51:05.836530 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:05.836321 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" podUID="d2656fde-164f-40ec-845f-ad9c08091597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 02 23:51:06.841302 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:06.841269 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" Nov 02 23:51:06.841760 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:06.841738 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" Nov 02 23:51:06.842714 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:06.842687 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" Nov 02 23:51:09.884358 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:51:09.884319 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:51:29.642383 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:29.642330 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-28f82c/auth-test-model-28f82c-kserve-65859bc8d9-vdlv4"] Nov 02 23:51:33.781036 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:33.780998 5076 generic.go:356] "Generic (PLEG): container finished" podID="7947bd91-9f2d-44dc-9e12-d3c8f3744ab0" containerID="e7a104e2d6299ccea44e62b5db27a57544f56dc0d2828eb158201b72a8c2a40d" exitCode=0 Nov 02 23:51:33.781515 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:33.781067 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" event={"ID":"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0","Type":"ContainerDied","Data":"e7a104e2d6299ccea44e62b5db27a57544f56dc0d2828eb158201b72a8c2a40d"} Nov 02 23:51:33.781515 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:33.781460 5076 scope.go:122] "RemoveContainer" containerID="e7a104e2d6299ccea44e62b5db27a57544f56dc0d2828eb158201b72a8c2a40d" Nov 02 23:51:33.783064 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:33.783039 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:51:33.783129 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:33.783112 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:51:33.783165 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:33.783146 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:51:34.790303 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:34.790263 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" event={"ID":"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0","Type":"ContainerStarted","Data":"7837c3ff776b93b2944c856efbbae2b2d6af94b6ecf8c5f8db0dbe2664422f44"} Nov 02 23:51:35.569024 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:35.568975 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" Nov 02 23:51:35.569366 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:35.569183 5076 patch_prober.go:29] interesting pod/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 02 23:51:35.569551 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:35.569519 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" podUID="7947bd91-9f2d-44dc-9e12-d3c8f3744ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 02 23:51:36.569164 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:36.569133 5076 patch_prober.go:29] interesting pod/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 02 23:51:36.569564 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:36.569184 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" podUID="7947bd91-9f2d-44dc-9e12-d3c8f3744ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 02 23:51:37.569350 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:37.569313 5076 patch_prober.go:29] interesting pod/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 02 23:51:37.569754 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:37.569386 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" podUID="7947bd91-9f2d-44dc-9e12-d3c8f3744ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 02 23:51:38.569338 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:38.569300 5076 patch_prober.go:29] interesting pod/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 02 23:51:38.569541 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:38.569354 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" podUID="7947bd91-9f2d-44dc-9e12-d3c8f3744ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 02 23:51:39.569367 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:39.569330 5076 patch_prober.go:29] interesting pod/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 02 23:51:39.569554 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:39.569387 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" podUID="7947bd91-9f2d-44dc-9e12-d3c8f3744ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 02 23:51:39.886055 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:51:39.885972 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:51:40.573811 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:40.573783 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" Nov 02 23:51:40.574264 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:40.574227 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" Nov 02 23:51:40.575111 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:40.575086 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" Nov 02 23:51:43.685937 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:43.685844 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm"] Nov 02 23:51:43.689099 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:43.689080 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm" Nov 02 23:51:43.720472 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:43.720439 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfl88\" (UniqueName: \"kubernetes.io/projected/efd74f02-817e-4143-a98d-6d82d8fc3e2b-kube-api-access-kfl88\") pod \"maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm\" (UID: \"efd74f02-817e-4143-a98d-6d82d8fc3e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm" Nov 02 23:51:43.720625 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:43.720569 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efd74f02-817e-4143-a98d-6d82d8fc3e2b-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm\" (UID: \"efd74f02-817e-4143-a98d-6d82d8fc3e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm" Nov 02 23:51:43.720681 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:43.720666 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efd74f02-817e-4143-a98d-6d82d8fc3e2b-metrics-certs\") pod \"maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm\" (UID: \"efd74f02-817e-4143-a98d-6d82d8fc3e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm" Nov 02 23:51:43.720752 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:43.720739 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efd74f02-817e-4143-a98d-6d82d8fc3e2b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm\" (UID: \"efd74f02-817e-4143-a98d-6d82d8fc3e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm" Nov 02 23:51:43.742801 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:43.742769 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm"] Nov 02 23:51:43.786706 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:43.786671 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-e65845-serving-cert\"" Nov 02 23:51:43.822270 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:43.822234 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efd74f02-817e-4143-a98d-6d82d8fc3e2b-metrics-certs\") pod \"maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm\" (UID: \"efd74f02-817e-4143-a98d-6d82d8fc3e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm" Nov 02 23:51:43.822483 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:43.822353 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efd74f02-817e-4143-a98d-6d82d8fc3e2b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm\" (UID: \"efd74f02-817e-4143-a98d-6d82d8fc3e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm" Nov 02 23:51:43.822483 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:43.822391 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfl88\" (UniqueName: \"kubernetes.io/projected/efd74f02-817e-4143-a98d-6d82d8fc3e2b-kube-api-access-kfl88\") pod \"maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm\" (UID: \"efd74f02-817e-4143-a98d-6d82d8fc3e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm" Nov 02 23:51:43.822483 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:51:43.822456 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-e65845-serving-cert: secret "maas-api-e2e-extmod-a-e65845-serving-cert" not found Nov 02 23:51:43.822483 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:43.822478 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efd74f02-817e-4143-a98d-6d82d8fc3e2b-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm\" (UID: \"efd74f02-817e-4143-a98d-6d82d8fc3e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm" Nov 02 23:51:43.822979 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:51:43.822575 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efd74f02-817e-4143-a98d-6d82d8fc3e2b-maas-api-tls podName:efd74f02-817e-4143-a98d-6d82d8fc3e2b nodeName:}" failed. No retries permitted until 2026-11-02 23:51:44.322553243 +0000 UTC m=+3553.291085010 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/efd74f02-817e-4143-a98d-6d82d8fc3e2b-maas-api-tls") pod "maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm" (UID: "efd74f02-817e-4143-a98d-6d82d8fc3e2b") : secret "maas-api-e2e-extmod-a-e65845-serving-cert" not found Nov 02 23:51:43.823207 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:43.823187 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efd74f02-817e-4143-a98d-6d82d8fc3e2b-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm\" (UID: \"efd74f02-817e-4143-a98d-6d82d8fc3e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm" Nov 02 23:51:43.837342 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:43.837311 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efd74f02-817e-4143-a98d-6d82d8fc3e2b-metrics-certs\") pod \"maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm\" (UID: \"efd74f02-817e-4143-a98d-6d82d8fc3e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm" Nov 02 23:51:43.894206 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:43.894170 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfl88\" (UniqueName: \"kubernetes.io/projected/efd74f02-817e-4143-a98d-6d82d8fc3e2b-kube-api-access-kfl88\") pod \"maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm\" (UID: \"efd74f02-817e-4143-a98d-6d82d8fc3e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm" Nov 02 23:51:44.330642 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:44.330590 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efd74f02-817e-4143-a98d-6d82d8fc3e2b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm\" (UID: \"efd74f02-817e-4143-a98d-6d82d8fc3e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm" Nov 02 23:51:44.334407 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:44.334374 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efd74f02-817e-4143-a98d-6d82d8fc3e2b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm\" (UID: \"efd74f02-817e-4143-a98d-6d82d8fc3e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm" Nov 02 23:51:44.602230 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:44.602138 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm" Nov 02 23:51:44.672348 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:44.672155 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-28f82c/auth-test-model-28f82c-kserve-65859bc8d9-vdlv4" podUID="20d23100-5ee8-40f1-b654-00755c3ab573" containerName="main" containerID="cri-o://36c7db146bfcbfa7827540766343540847c6dd61fa420d95e7910ecd22d7532f" gracePeriod=45 Nov 02 23:51:44.885517 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:44.885426 5076 generic.go:356] "Generic (PLEG): container finished" podID="20d23100-5ee8-40f1-b654-00755c3ab573" containerID="36c7db146bfcbfa7827540766343540847c6dd61fa420d95e7910ecd22d7532f" exitCode=0 Nov 02 23:51:44.885517 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:44.885465 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-28f82c/auth-test-model-28f82c-kserve-65859bc8d9-vdlv4" event={"ID":"20d23100-5ee8-40f1-b654-00755c3ab573","Type":"ContainerDied","Data":"36c7db146bfcbfa7827540766343540847c6dd61fa420d95e7910ecd22d7532f"} Nov 02 23:51:44.947647 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:44.947619 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-28f82c/auth-test-model-28f82c-kserve-65859bc8d9-vdlv4" Nov 02 23:51:44.972298 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:44.972260 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm"] Nov 02 23:51:44.975915 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:51:44.975858 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefd74f02_817e_4143_a98d_6d82d8fc3e2b.slice/crio-c543fe354231e4c2efca8272e894fac434c965d93405fabaf6e1921c0b6defbb WatchSource:0}: Error finding container c543fe354231e4c2efca8272e894fac434c965d93405fabaf6e1921c0b6defbb: Status 404 returned error can't find the container with id c543fe354231e4c2efca8272e894fac434c965d93405fabaf6e1921c0b6defbb Nov 02 23:51:45.037796 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:45.037765 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20d23100-5ee8-40f1-b654-00755c3ab573-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20d23100-5ee8-40f1-b654-00755c3ab573-model-cache\") pod \"20d23100-5ee8-40f1-b654-00755c3ab573\" (UID: \"20d23100-5ee8-40f1-b654-00755c3ab573\") " Nov 02 23:51:45.038026 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:45.037818 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20d23100-5ee8-40f1-b654-00755c3ab573-tls-certs\" (UniqueName: \"kubernetes.io/secret/20d23100-5ee8-40f1-b654-00755c3ab573-tls-certs\") pod \"20d23100-5ee8-40f1-b654-00755c3ab573\" (UID: \"20d23100-5ee8-40f1-b654-00755c3ab573\") " Nov 02 23:51:45.038026 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:45.037854 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20d23100-5ee8-40f1-b654-00755c3ab573-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20d23100-5ee8-40f1-b654-00755c3ab573-tmp-dir\") pod \"20d23100-5ee8-40f1-b654-00755c3ab573\" (UID: \"20d23100-5ee8-40f1-b654-00755c3ab573\") " Nov 02 23:51:45.038026 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:45.037878 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20d23100-5ee8-40f1-b654-00755c3ab573-dshm\" (UniqueName: \"kubernetes.io/empty-dir/20d23100-5ee8-40f1-b654-00755c3ab573-dshm\") pod \"20d23100-5ee8-40f1-b654-00755c3ab573\" (UID: \"20d23100-5ee8-40f1-b654-00755c3ab573\") " Nov 02 23:51:45.038026 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:45.037988 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20d23100-5ee8-40f1-b654-00755c3ab573-home\" (UniqueName: \"kubernetes.io/empty-dir/20d23100-5ee8-40f1-b654-00755c3ab573-home\") pod \"20d23100-5ee8-40f1-b654-00755c3ab573\" (UID: \"20d23100-5ee8-40f1-b654-00755c3ab573\") " Nov 02 23:51:45.038265 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:45.038079 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d23100-5ee8-40f1-b654-00755c3ab573-model-cache" pod "20d23100-5ee8-40f1-b654-00755c3ab573" (UID: "20d23100-5ee8-40f1-b654-00755c3ab573"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:51:45.038265 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:45.038173 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20d23100-5ee8-40f1-b654-00755c3ab573-kube-api-access-tdjzw\" (UniqueName: \"kubernetes.io/projected/20d23100-5ee8-40f1-b654-00755c3ab573-kube-api-access-tdjzw\") pod \"20d23100-5ee8-40f1-b654-00755c3ab573\" (UID: \"20d23100-5ee8-40f1-b654-00755c3ab573\") " Nov 02 23:51:45.038265 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:45.038213 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d23100-5ee8-40f1-b654-00755c3ab573-tmp-dir" pod "20d23100-5ee8-40f1-b654-00755c3ab573" (UID: "20d23100-5ee8-40f1-b654-00755c3ab573"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:51:45.038456 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:45.038322 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d23100-5ee8-40f1-b654-00755c3ab573-home" pod "20d23100-5ee8-40f1-b654-00755c3ab573" (UID: "20d23100-5ee8-40f1-b654-00755c3ab573"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:51:45.038763 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:45.038690 5076 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20d23100-5ee8-40f1-b654-00755c3ab573-model-cache\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:51:45.038763 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:45.038718 5076 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20d23100-5ee8-40f1-b654-00755c3ab573-tmp-dir\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:51:45.038763 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:45.038734 5076 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20d23100-5ee8-40f1-b654-00755c3ab573-home\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:51:45.040467 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:45.040444 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d23100-5ee8-40f1-b654-00755c3ab573-dshm" pod "20d23100-5ee8-40f1-b654-00755c3ab573" (UID: "20d23100-5ee8-40f1-b654-00755c3ab573"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:51:45.040570 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:45.040521 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20d23100-5ee8-40f1-b654-00755c3ab573-tls-certs" pod "20d23100-5ee8-40f1-b654-00755c3ab573" (UID: "20d23100-5ee8-40f1-b654-00755c3ab573"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:51:45.040570 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:45.040563 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d23100-5ee8-40f1-b654-00755c3ab573-kube-api-access-tdjzw" pod "20d23100-5ee8-40f1-b654-00755c3ab573" (UID: "20d23100-5ee8-40f1-b654-00755c3ab573"). InnerVolumeSpecName "kube-api-access-tdjzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:51:45.139756 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:45.139669 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdjzw\" (UniqueName: \"kubernetes.io/projected/20d23100-5ee8-40f1-b654-00755c3ab573-kube-api-access-tdjzw\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:51:45.139756 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:45.139702 5076 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20d23100-5ee8-40f1-b654-00755c3ab573-tls-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:51:45.139756 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:45.139713 5076 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20d23100-5ee8-40f1-b654-00755c3ab573-dshm\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:51:45.894614 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:45.894586 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-28f82c/auth-test-model-28f82c-kserve-65859bc8d9-vdlv4" Nov 02 23:51:45.894614 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:45.894603 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-28f82c/auth-test-model-28f82c-kserve-65859bc8d9-vdlv4" event={"ID":"20d23100-5ee8-40f1-b654-00755c3ab573","Type":"ContainerDied","Data":"6e0408a506d3d430f890bd40266b5f670a8cb9bd37c0d477a4a7019d2cb2a0e6"} Nov 02 23:51:45.895203 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:45.894653 5076 scope.go:122] "RemoveContainer" containerID="36c7db146bfcbfa7827540766343540847c6dd61fa420d95e7910ecd22d7532f" Nov 02 23:51:45.896153 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:45.896124 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm" event={"ID":"efd74f02-817e-4143-a98d-6d82d8fc3e2b","Type":"ContainerStarted","Data":"c543fe354231e4c2efca8272e894fac434c965d93405fabaf6e1921c0b6defbb"} Nov 02 23:51:46.050598 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:46.050548 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-28f82c/auth-test-model-28f82c-kserve-65859bc8d9-vdlv4"] Nov 02 23:51:46.054440 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:46.054411 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-28f82c/auth-test-model-28f82c-kserve-65859bc8d9-vdlv4"] Nov 02 23:51:46.906531 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:46.906492 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm" event={"ID":"efd74f02-817e-4143-a98d-6d82d8fc3e2b","Type":"ContainerStarted","Data":"1f54a180dd7da9d2d29c8b0b69c7566702577d5436b03ade0b77ce649978a751"} Nov 02 23:51:46.906963 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:46.906620 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm" Nov 02 23:51:47.016867 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:47.016782 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm" podStartSLOduration=3.141376845 podStartE2EDuration="4.016767656s" totalImagesPullingTime="875.390811ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:51:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:51:47.015192278 +0000 UTC m=+3555.983724062" watchObservedRunningTime="2026-11-02 23:51:47.016767656 +0000 UTC m=+3555.985299440" Nov 02 23:51:47.159354 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:47.159250 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20d23100-5ee8-40f1-b654-00755c3ab573" path="/var/lib/kubelet/pods/20d23100-5ee8-40f1-b654-00755c3ab573/volumes" Nov 02 23:51:52.917338 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:51:52.917308 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm" Nov 02 23:52:09.885232 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:52:09.885196 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:52:22.837546 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:22.837512 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2"] Nov 02 23:52:22.838331 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:22.838307 5076 cpu_manager.go:414] "containerMap: removing container" podUID="20d23100-5ee8-40f1-b654-00755c3ab573" containerName="main" Nov 02 23:52:22.838526 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:22.838493 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20d23100-5ee8-40f1-b654-00755c3ab573" containerName="main" Nov 02 23:52:22.838724 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:22.838701 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="20d23100-5ee8-40f1-b654-00755c3ab573" containerName="main" Nov 02 23:52:22.845273 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:22.845251 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" Nov 02 23:52:22.874647 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:22.874613 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2"] Nov 02 23:52:22.927913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:22.926189 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-4b63d8-openshift-default-dockercfg-xkg2f\"" Nov 02 23:52:23.043316 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:23.038965 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fa13122-f046-4621-bf87-d124725da22f-istio-envoy\") pod \"e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2\" (UID: \"9fa13122-f046-4621-bf87-d124725da22f\") " pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" Nov 02 23:52:23.043727 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:23.043701 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fa13122-f046-4621-bf87-d124725da22f-workload-socket\") pod \"e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2\" (UID: \"9fa13122-f046-4621-bf87-d124725da22f\") " pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" Nov 02 23:52:23.044709 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:23.044674 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fa13122-f046-4621-bf87-d124725da22f-credential-socket\") pod \"e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2\" (UID: \"9fa13122-f046-4621-bf87-d124725da22f\") " pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" Nov 02 23:52:23.044917 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:23.044880 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fa13122-f046-4621-bf87-d124725da22f-workload-certs\") pod \"e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2\" (UID: \"9fa13122-f046-4621-bf87-d124725da22f\") " pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" Nov 02 23:52:23.045050 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:23.045035 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fa13122-f046-4621-bf87-d124725da22f-istio-token\") pod \"e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2\" (UID: \"9fa13122-f046-4621-bf87-d124725da22f\") " pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" Nov 02 23:52:23.045220 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:23.045203 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fa13122-f046-4621-bf87-d124725da22f-istio-data\") pod \"e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2\" (UID: \"9fa13122-f046-4621-bf87-d124725da22f\") " pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" Nov 02 23:52:23.045967 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:23.045945 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj77t\" (UniqueName: \"kubernetes.io/projected/9fa13122-f046-4621-bf87-d124725da22f-kube-api-access-wj77t\") pod \"e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2\" (UID: \"9fa13122-f046-4621-bf87-d124725da22f\") " pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" Nov 02 23:52:23.046126 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:23.046108 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fa13122-f046-4621-bf87-d124725da22f-istio-podinfo\") pod \"e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2\" (UID: \"9fa13122-f046-4621-bf87-d124725da22f\") " pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" Nov 02 23:52:23.046306 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:23.046288 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fa13122-f046-4621-bf87-d124725da22f-istiod-ca-cert\") pod \"e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2\" (UID: \"9fa13122-f046-4621-bf87-d124725da22f\") " pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" Nov 02 23:52:23.147663 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:23.147584 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fa13122-f046-4621-bf87-d124725da22f-istiod-ca-cert\") pod \"e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2\" (UID: \"9fa13122-f046-4621-bf87-d124725da22f\") " pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" Nov 02 23:52:23.147663 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:23.147656 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fa13122-f046-4621-bf87-d124725da22f-istio-envoy\") pod \"e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2\" (UID: \"9fa13122-f046-4621-bf87-d124725da22f\") " pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" Nov 02 23:52:23.147851 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:23.147744 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fa13122-f046-4621-bf87-d124725da22f-workload-socket\") pod \"e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2\" (UID: \"9fa13122-f046-4621-bf87-d124725da22f\") " pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" Nov 02 23:52:23.147851 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:23.147766 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fa13122-f046-4621-bf87-d124725da22f-credential-socket\") pod \"e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2\" (UID: \"9fa13122-f046-4621-bf87-d124725da22f\") " pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" Nov 02 23:52:23.147851 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:23.147787 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fa13122-f046-4621-bf87-d124725da22f-workload-certs\") pod \"e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2\" (UID: \"9fa13122-f046-4621-bf87-d124725da22f\") " pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" Nov 02 23:52:23.147851 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:23.147808 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fa13122-f046-4621-bf87-d124725da22f-istio-token\") pod \"e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2\" (UID: \"9fa13122-f046-4621-bf87-d124725da22f\") " pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" Nov 02 23:52:23.147851 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:23.147837 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fa13122-f046-4621-bf87-d124725da22f-istio-data\") pod \"e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2\" (UID: \"9fa13122-f046-4621-bf87-d124725da22f\") " pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" Nov 02 23:52:23.148132 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:23.147856 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj77t\" (UniqueName: \"kubernetes.io/projected/9fa13122-f046-4621-bf87-d124725da22f-kube-api-access-wj77t\") pod \"e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2\" (UID: \"9fa13122-f046-4621-bf87-d124725da22f\") " pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" Nov 02 23:52:23.148132 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:23.147877 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fa13122-f046-4621-bf87-d124725da22f-istio-podinfo\") pod \"e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2\" (UID: \"9fa13122-f046-4621-bf87-d124725da22f\") " pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" Nov 02 23:52:23.149439 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:23.148322 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fa13122-f046-4621-bf87-d124725da22f-credential-socket\") pod \"e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2\" (UID: \"9fa13122-f046-4621-bf87-d124725da22f\") " pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" Nov 02 23:52:23.149439 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:23.148738 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fa13122-f046-4621-bf87-d124725da22f-workload-certs\") pod \"e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2\" (UID: \"9fa13122-f046-4621-bf87-d124725da22f\") " pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" Nov 02 23:52:23.149439 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:23.149084 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fa13122-f046-4621-bf87-d124725da22f-istio-data\") pod \"e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2\" (UID: \"9fa13122-f046-4621-bf87-d124725da22f\") " pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" Nov 02 23:52:23.149439 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:23.149352 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fa13122-f046-4621-bf87-d124725da22f-istiod-ca-cert\") pod \"e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2\" (UID: \"9fa13122-f046-4621-bf87-d124725da22f\") " pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" Nov 02 23:52:23.150451 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:23.149513 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fa13122-f046-4621-bf87-d124725da22f-workload-socket\") pod \"e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2\" (UID: \"9fa13122-f046-4621-bf87-d124725da22f\") " pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" Nov 02 23:52:23.151334 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:23.151311 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fa13122-f046-4621-bf87-d124725da22f-istio-envoy\") pod \"e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2\" (UID: \"9fa13122-f046-4621-bf87-d124725da22f\") " pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" Nov 02 23:52:23.152145 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:23.152124 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fa13122-f046-4621-bf87-d124725da22f-istio-podinfo\") pod \"e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2\" (UID: \"9fa13122-f046-4621-bf87-d124725da22f\") " pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" Nov 02 23:52:23.206849 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:23.206811 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fa13122-f046-4621-bf87-d124725da22f-istio-token\") pod \"e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2\" (UID: \"9fa13122-f046-4621-bf87-d124725da22f\") " pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" Nov 02 23:52:23.251085 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:23.251052 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj77t\" (UniqueName: \"kubernetes.io/projected/9fa13122-f046-4621-bf87-d124725da22f-kube-api-access-wj77t\") pod \"e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2\" (UID: \"9fa13122-f046-4621-bf87-d124725da22f\") " pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" Nov 02 23:52:23.460693 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:23.460652 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" Nov 02 23:52:23.902659 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:23.902628 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2"] Nov 02 23:52:23.903259 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:52:23.903236 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fa13122_f046_4621_bf87_d124725da22f.slice/crio-2b35f131bd152261edf77dbd9e3c0023b7bd0c27a8c72c8bd93b08ab17ecf034 WatchSource:0}: Error finding container 2b35f131bd152261edf77dbd9e3c0023b7bd0c27a8c72c8bd93b08ab17ecf034: Status 404 returned error can't find the container with id 2b35f131bd152261edf77dbd9e3c0023b7bd0c27a8c72c8bd93b08ab17ecf034 Nov 02 23:52:23.905926 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:23.905864 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:52:23.906007 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:23.905964 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:52:23.906007 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:23.905995 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:52:24.175931 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:24.175813 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" event={"ID":"9fa13122-f046-4621-bf87-d124725da22f","Type":"ContainerStarted","Data":"09fe8c125532ecb19785795e189dfc0e452a67429777d199981bd9c26eb2e8cb"} Nov 02 23:52:24.175931 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:24.175870 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" event={"ID":"9fa13122-f046-4621-bf87-d124725da22f","Type":"ContainerStarted","Data":"2b35f131bd152261edf77dbd9e3c0023b7bd0c27a8c72c8bd93b08ab17ecf034"} Nov 02 23:52:24.290847 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:24.290790 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" podStartSLOduration=2.290774939 podStartE2EDuration="2.290774939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:52:24.289217754 +0000 UTC m=+3593.257749549" watchObservedRunningTime="2026-11-02 23:52:24.290774939 +0000 UTC m=+3593.259306722" Nov 02 23:52:24.461375 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:24.461340 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" Nov 02 23:52:25.478067 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:25.478034 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" Nov 02 23:52:26.189222 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:26.189186 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" Nov 02 23:52:26.190853 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:26.190826 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" Nov 02 23:52:27.780770 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:27.780738 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7"] Nov 02 23:52:27.785308 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:27.785289 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7" Nov 02 23:52:27.808968 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:27.808940 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7\" (UID: \"73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7" Nov 02 23:52:27.809096 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:27.808983 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b-metrics-certs\") pod \"maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7\" (UID: \"73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7" Nov 02 23:52:27.809153 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:27.809126 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzbk4\" (UniqueName: \"kubernetes.io/projected/73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b-kube-api-access-pzbk4\") pod \"maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7\" (UID: \"73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7" Nov 02 23:52:27.809257 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:27.809242 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7\" (UID: \"73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7" Nov 02 23:52:27.834808 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:27.834776 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-f2df33-serving-cert\"" Nov 02 23:52:27.888052 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:27.888012 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7"] Nov 02 23:52:27.910432 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:27.910392 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7\" (UID: \"73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7" Nov 02 23:52:27.910623 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:27.910444 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b-metrics-certs\") pod \"maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7\" (UID: \"73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7" Nov 02 23:52:27.910623 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:27.910532 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzbk4\" (UniqueName: \"kubernetes.io/projected/73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b-kube-api-access-pzbk4\") pod \"maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7\" (UID: \"73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7" Nov 02 23:52:27.910623 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:52:27.910574 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-f2df33-serving-cert: secret "maas-api-e2e-extmod-a-f2df33-serving-cert" not found Nov 02 23:52:27.910873 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:27.910650 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7\" (UID: \"73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7" Nov 02 23:52:27.910873 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:52:27.910679 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b-maas-api-tls podName:73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b nodeName:}" failed. No retries permitted until 2026-11-02 23:52:28.410656181 +0000 UTC m=+3597.379187978 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b-maas-api-tls") pod "maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7" (UID: "73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b") : secret "maas-api-e2e-extmod-a-f2df33-serving-cert" not found Nov 02 23:52:27.911431 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:27.911402 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7\" (UID: \"73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7" Nov 02 23:52:27.913312 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:27.913288 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b-metrics-certs\") pod \"maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7\" (UID: \"73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7" Nov 02 23:52:27.966879 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:27.966835 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzbk4\" (UniqueName: \"kubernetes.io/projected/73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b-kube-api-access-pzbk4\") pod \"maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7\" (UID: \"73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7" Nov 02 23:52:28.267064 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:28.267016 5076 patch_prober.go:29] interesting pod/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:52:28.267260 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:28.267080 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" podUID="000f0ea7-6931-4670-9d4f-814c072405c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:52:28.430224 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:28.430190 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7\" (UID: \"73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7" Nov 02 23:52:28.432829 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:28.432615 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7\" (UID: \"73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7" Nov 02 23:52:28.702701 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:28.702656 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7" Nov 02 23:52:29.074203 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:29.074172 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7"] Nov 02 23:52:29.075050 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:52:29.075021 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73564eaf_3199_4a0b_8fc2_cbc7ee72eb9b.slice/crio-efb23a83712cbf4678fddd0c5a48fae6c89595a65c4dc715631da3a3eaa334f5 WatchSource:0}: Error finding container efb23a83712cbf4678fddd0c5a48fae6c89595a65c4dc715631da3a3eaa334f5: Status 404 returned error can't find the container with id efb23a83712cbf4678fddd0c5a48fae6c89595a65c4dc715631da3a3eaa334f5 Nov 02 23:52:29.219858 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:29.219826 5076 generic.go:356] "Generic (PLEG): container finished" podID="000f0ea7-6931-4670-9d4f-814c072405c9" containerID="7778c67442e41bf95c06a3aa3e62061243a047374935a420da52367cb9445e6c" exitCode=0 Nov 02 23:52:29.220052 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:29.219905 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" event={"ID":"000f0ea7-6931-4670-9d4f-814c072405c9","Type":"ContainerDied","Data":"7778c67442e41bf95c06a3aa3e62061243a047374935a420da52367cb9445e6c"} Nov 02 23:52:29.220381 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:29.220355 5076 scope.go:122] "RemoveContainer" containerID="7778c67442e41bf95c06a3aa3e62061243a047374935a420da52367cb9445e6c" Nov 02 23:52:29.221224 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:29.221200 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7" event={"ID":"73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b","Type":"ContainerStarted","Data":"efb23a83712cbf4678fddd0c5a48fae6c89595a65c4dc715631da3a3eaa334f5"} Nov 02 23:52:29.222001 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:29.221975 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:52:29.222058 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:29.222043 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:52:29.222095 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:29.222077 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:52:30.241219 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:30.241121 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" event={"ID":"000f0ea7-6931-4670-9d4f-814c072405c9","Type":"ContainerStarted","Data":"a18ded61a21194f71ff35fc49cec0013c423eaf4133ebc338e45f095ac24f296"} Nov 02 23:52:30.242675 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:30.242652 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7" event={"ID":"73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b","Type":"ContainerStarted","Data":"a47577f1e5abf31903a850e979cf2f637c0b1a71294d3d7f0376344c9f79f99e"} Nov 02 23:52:30.242802 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:30.242770 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7" Nov 02 23:52:30.473064 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:30.473010 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7" podStartSLOduration=2.560399763 podStartE2EDuration="3.472995108s" totalImagesPullingTime="912.595345ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:52:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:52:30.471148869 +0000 UTC m=+3599.439680653" watchObservedRunningTime="2026-11-02 23:52:30.472995108 +0000 UTC m=+3599.441526889" Nov 02 23:52:31.159011 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:31.158291 5076 patch_prober.go:29] interesting pod/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 02 23:52:31.159011 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:31.158349 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" podUID="000f0ea7-6931-4670-9d4f-814c072405c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 02 23:52:31.165079 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:31.165051 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" Nov 02 23:52:31.976396 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:31.976367 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 02 23:52:32.002464 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:32.002434 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 02 23:52:32.140097 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:32.140060 5076 patch_prober.go:29] interesting pod/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 02 23:52:32.140271 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:32.140119 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" podUID="000f0ea7-6931-4670-9d4f-814c072405c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 02 23:52:33.141075 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:33.141037 5076 patch_prober.go:29] interesting pod/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 02 23:52:33.141580 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:33.141085 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" podUID="000f0ea7-6931-4670-9d4f-814c072405c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 02 23:52:34.140089 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:34.140052 5076 patch_prober.go:29] interesting pod/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 02 23:52:34.140294 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:34.140104 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" podUID="000f0ea7-6931-4670-9d4f-814c072405c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 02 23:52:35.140453 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:35.140415 5076 patch_prober.go:29] interesting pod/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 02 23:52:35.140864 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:35.140470 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" podUID="000f0ea7-6931-4670-9d4f-814c072405c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 02 23:52:36.144602 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:36.144569 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" Nov 02 23:52:36.145097 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:36.145061 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" Nov 02 23:52:36.145689 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:36.145669 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" Nov 02 23:52:36.304647 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:52:36.304617 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7" Nov 02 23:52:39.885018 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:52:39.884984 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:53:09.884650 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:53:09.884611 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:53:24.670946 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:24.670877 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj"] Nov 02 23:53:24.675350 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:24.675328 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" Nov 02 23:53:24.753990 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:24.753953 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj"] Nov 02 23:53:24.774967 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:24.774938 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-a8a1dd-openshift-default-dockercfg-rwn8b\"" Nov 02 23:53:24.786536 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:24.786506 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-istiod-ca-cert\") pod \"e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj\" (UID: \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\") " pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" Nov 02 23:53:24.786677 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:24.786551 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-credential-socket\") pod \"e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj\" (UID: \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\") " pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" Nov 02 23:53:24.786677 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:24.786640 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-istio-envoy\") pod \"e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj\" (UID: \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\") " pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" Nov 02 23:53:24.786780 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:24.786760 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-workload-socket\") pod \"e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj\" (UID: \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\") " pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" Nov 02 23:53:24.786818 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:24.786803 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-istio-podinfo\") pod \"e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj\" (UID: \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\") " pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" Nov 02 23:53:24.786857 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:24.786825 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-istio-token\") pod \"e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj\" (UID: \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\") " pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" Nov 02 23:53:24.787057 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:24.787025 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jv79\" (UniqueName: \"kubernetes.io/projected/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-kube-api-access-2jv79\") pod \"e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj\" (UID: \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\") " pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" Nov 02 23:53:24.787143 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:24.787073 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-istio-data\") pod \"e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj\" (UID: \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\") " pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" Nov 02 23:53:24.787183 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:24.787155 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-workload-certs\") pod \"e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj\" (UID: \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\") " pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" Nov 02 23:53:24.908149 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:24.908107 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-istiod-ca-cert\") pod \"e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj\" (UID: \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\") " pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" Nov 02 23:53:24.909400 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:24.909375 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-credential-socket\") pod \"e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj\" (UID: \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\") " pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" Nov 02 23:53:24.909576 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:24.909561 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-istio-envoy\") pod \"e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj\" (UID: \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\") " pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" Nov 02 23:53:24.909745 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:24.909733 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-workload-socket\") pod \"e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj\" (UID: \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\") " pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" Nov 02 23:53:24.909844 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:24.909830 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-istio-podinfo\") pod \"e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj\" (UID: \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\") " pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" Nov 02 23:53:24.909999 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:24.909981 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-istio-token\") pod \"e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj\" (UID: \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\") " pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" Nov 02 23:53:24.910203 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:24.910184 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jv79\" (UniqueName: \"kubernetes.io/projected/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-kube-api-access-2jv79\") pod \"e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj\" (UID: \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\") " pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" Nov 02 23:53:24.910331 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:24.910315 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-istio-data\") pod \"e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj\" (UID: \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\") " pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" Nov 02 23:53:24.910458 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:24.910445 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-workload-certs\") pod \"e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj\" (UID: \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\") " pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" Nov 02 23:53:24.923962 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:24.921288 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-workload-socket\") pod \"e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj\" (UID: \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\") " pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" Nov 02 23:53:24.923962 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:24.909307 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-istiod-ca-cert\") pod \"e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj\" (UID: \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\") " pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" Nov 02 23:53:24.923962 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:24.921589 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-credential-socket\") pod \"e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj\" (UID: \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\") " pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" Nov 02 23:53:24.925076 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:24.925039 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-istio-data\") pod \"e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj\" (UID: \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\") " pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" Nov 02 23:53:24.925347 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:24.925324 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-workload-certs\") pod \"e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj\" (UID: \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\") " pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" Nov 02 23:53:24.949231 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:24.949197 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-istio-envoy\") pod \"e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj\" (UID: \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\") " pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" Nov 02 23:53:24.952569 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:24.952502 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-istio-podinfo\") pod \"e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj\" (UID: \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\") " pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" Nov 02 23:53:24.984659 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:24.984590 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-istio-token\") pod \"e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj\" (UID: \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\") " pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" Nov 02 23:53:24.987033 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:24.986949 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jv79\" (UniqueName: \"kubernetes.io/projected/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-kube-api-access-2jv79\") pod \"e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj\" (UID: \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\") " pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" Nov 02 23:53:24.991805 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:24.991722 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" Nov 02 23:53:25.439504 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:25.439460 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj"] Nov 02 23:53:25.472067 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:53:25.472022 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56fa477d_d2c0_4a6b_a3b4_51cdf5420002.slice/crio-9bfc5fc56bcd34bc61a370e3adfccc4f9d144e3ad5932994831fce33bc05cfde WatchSource:0}: Error finding container 9bfc5fc56bcd34bc61a370e3adfccc4f9d144e3ad5932994831fce33bc05cfde: Status 404 returned error can't find the container with id 9bfc5fc56bcd34bc61a370e3adfccc4f9d144e3ad5932994831fce33bc05cfde Nov 02 23:53:25.475400 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:25.475362 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:53:25.475482 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:25.475447 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:53:25.475520 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:25.475490 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:53:25.639349 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:25.639306 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" event={"ID":"56fa477d-d2c0-4a6b-a3b4-51cdf5420002","Type":"ContainerStarted","Data":"1727703d39998e9704197b736d617afd642a267537407ef2852dd65c691ee0d3"} Nov 02 23:53:25.639561 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:25.639373 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" event={"ID":"56fa477d-d2c0-4a6b-a3b4-51cdf5420002","Type":"ContainerStarted","Data":"9bfc5fc56bcd34bc61a370e3adfccc4f9d144e3ad5932994831fce33bc05cfde"} Nov 02 23:53:25.752550 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:25.752430 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" podStartSLOduration=1.752411247 podStartE2EDuration="1.752411247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:53:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:53:25.750668053 +0000 UTC m=+3654.719199836" watchObservedRunningTime="2026-11-02 23:53:25.752411247 +0000 UTC m=+3654.720943031" Nov 02 23:53:25.992813 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:25.992771 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" Nov 02 23:53:26.999029 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:26.998993 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" Nov 02 23:53:27.654749 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:27.654721 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" Nov 02 23:53:27.655649 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:53:27.655630 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" Nov 02 23:53:39.883995 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:53:39.883954 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:54:01.167527 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:01.167484 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr"] Nov 02 23:54:01.172791 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:01.172774 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr" Nov 02 23:54:01.226192 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:01.226157 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr"] Nov 02 23:54:01.270075 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:01.270039 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-4b63d8-serving-cert\"" Nov 02 23:54:01.318207 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:01.318164 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a0ed3d-b728-4e61-81a2-1c118261632c-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr\" (UID: \"22a0ed3d-b728-4e61-81a2-1c118261632c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr" Nov 02 23:54:01.318404 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:01.318220 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22a0ed3d-b728-4e61-81a2-1c118261632c-maas-api-tls\") pod \"maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr\" (UID: \"22a0ed3d-b728-4e61-81a2-1c118261632c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr" Nov 02 23:54:01.318404 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:01.318337 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzcvh\" (UniqueName: \"kubernetes.io/projected/22a0ed3d-b728-4e61-81a2-1c118261632c-kube-api-access-mzcvh\") pod \"maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr\" (UID: \"22a0ed3d-b728-4e61-81a2-1c118261632c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr" Nov 02 23:54:01.318537 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:01.318414 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22a0ed3d-b728-4e61-81a2-1c118261632c-metrics-certs\") pod \"maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr\" (UID: \"22a0ed3d-b728-4e61-81a2-1c118261632c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr" Nov 02 23:54:01.423920 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:01.423810 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22a0ed3d-b728-4e61-81a2-1c118261632c-metrics-certs\") pod \"maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr\" (UID: \"22a0ed3d-b728-4e61-81a2-1c118261632c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr" Nov 02 23:54:01.430945 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:01.430779 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a0ed3d-b728-4e61-81a2-1c118261632c-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr\" (UID: \"22a0ed3d-b728-4e61-81a2-1c118261632c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr" Nov 02 23:54:01.430945 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:01.430842 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22a0ed3d-b728-4e61-81a2-1c118261632c-maas-api-tls\") pod \"maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr\" (UID: \"22a0ed3d-b728-4e61-81a2-1c118261632c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr" Nov 02 23:54:01.430945 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:01.430879 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzcvh\" (UniqueName: \"kubernetes.io/projected/22a0ed3d-b728-4e61-81a2-1c118261632c-kube-api-access-mzcvh\") pod \"maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr\" (UID: \"22a0ed3d-b728-4e61-81a2-1c118261632c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr" Nov 02 23:54:01.433588 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:01.429020 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22a0ed3d-b728-4e61-81a2-1c118261632c-metrics-certs\") pod \"maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr\" (UID: \"22a0ed3d-b728-4e61-81a2-1c118261632c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr" Nov 02 23:54:01.433588 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:01.432003 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a0ed3d-b728-4e61-81a2-1c118261632c-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr\" (UID: \"22a0ed3d-b728-4e61-81a2-1c118261632c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr" Nov 02 23:54:01.433588 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:54:01.432103 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-4b63d8-serving-cert: secret "maas-api-e2e-extmod-b-4b63d8-serving-cert" not found Nov 02 23:54:01.433588 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:54:01.432159 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22a0ed3d-b728-4e61-81a2-1c118261632c-maas-api-tls podName:22a0ed3d-b728-4e61-81a2-1c118261632c nodeName:}" failed. No retries permitted until 2026-11-02 23:54:01.932139972 +0000 UTC m=+3690.900671735 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/22a0ed3d-b728-4e61-81a2-1c118261632c-maas-api-tls") pod "maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr" (UID: "22a0ed3d-b728-4e61-81a2-1c118261632c") : secret "maas-api-e2e-extmod-b-4b63d8-serving-cert" not found Nov 02 23:54:01.485774 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:01.485745 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzcvh\" (UniqueName: \"kubernetes.io/projected/22a0ed3d-b728-4e61-81a2-1c118261632c-kube-api-access-mzcvh\") pod \"maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr\" (UID: \"22a0ed3d-b728-4e61-81a2-1c118261632c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr" Nov 02 23:54:01.937060 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:01.937026 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22a0ed3d-b728-4e61-81a2-1c118261632c-maas-api-tls\") pod \"maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr\" (UID: \"22a0ed3d-b728-4e61-81a2-1c118261632c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr" Nov 02 23:54:01.939421 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:01.939392 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22a0ed3d-b728-4e61-81a2-1c118261632c-maas-api-tls\") pod \"maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr\" (UID: \"22a0ed3d-b728-4e61-81a2-1c118261632c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr" Nov 02 23:54:02.086872 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:02.086834 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr" Nov 02 23:54:02.456043 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:02.456015 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr"] Nov 02 23:54:02.457312 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:54:02.457281 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a0ed3d_b728_4e61_81a2_1c118261632c.slice/crio-d5464b63232fdce8c4d876527147d763fdd96b700b1ec15966623b9f06b4e865 WatchSource:0}: Error finding container d5464b63232fdce8c4d876527147d763fdd96b700b1ec15966623b9f06b4e865: Status 404 returned error can't find the container with id d5464b63232fdce8c4d876527147d763fdd96b700b1ec15966623b9f06b4e865 Nov 02 23:54:02.901215 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:02.901128 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr" event={"ID":"22a0ed3d-b728-4e61-81a2-1c118261632c","Type":"ContainerStarted","Data":"d5464b63232fdce8c4d876527147d763fdd96b700b1ec15966623b9f06b4e865"} Nov 02 23:54:03.918701 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:03.918660 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr" event={"ID":"22a0ed3d-b728-4e61-81a2-1c118261632c","Type":"ContainerStarted","Data":"5d62c65bd3ca176e5e9dca96927ee6be92018b8167832742f05365cce1e751ce"} Nov 02 23:54:03.919157 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:03.918760 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr" Nov 02 23:54:04.029558 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:04.029494 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr" podStartSLOduration=2.161677057 podStartE2EDuration="3.029481577s" totalImagesPullingTime="867.80452ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:54:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:54:04.028366586 +0000 UTC m=+3692.996898383" watchObservedRunningTime="2026-11-02 23:54:04.029481577 +0000 UTC m=+3692.998013360" Nov 02 23:54:09.884636 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:54:09.884594 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:54:09.929443 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:09.929415 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr" Nov 02 23:54:36.178048 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:36.178014 5076 generic.go:356] "Generic (PLEG): container finished" podID="9fa13122-f046-4621-bf87-d124725da22f" containerID="09fe8c125532ecb19785795e189dfc0e452a67429777d199981bd9c26eb2e8cb" exitCode=0 Nov 02 23:54:36.178501 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:36.178088 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" event={"ID":"9fa13122-f046-4621-bf87-d124725da22f","Type":"ContainerDied","Data":"09fe8c125532ecb19785795e189dfc0e452a67429777d199981bd9c26eb2e8cb"} Nov 02 23:54:36.178593 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:36.178577 5076 scope.go:122] "RemoveContainer" containerID="09fe8c125532ecb19785795e189dfc0e452a67429777d199981bd9c26eb2e8cb" Nov 02 23:54:36.180364 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:36.180337 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:54:36.180436 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:36.180413 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:54:36.180474 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:36.180447 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:54:37.186771 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:37.186737 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" event={"ID":"9fa13122-f046-4621-bf87-d124725da22f","Type":"ContainerStarted","Data":"eb588c2e1e7d6a7f92ea7a9fea5d07fedd5840c8b3365b664c3d701f1b325b65"} Nov 02 23:54:37.461486 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:37.461403 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" Nov 02 23:54:37.462286 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:37.462262 5076 patch_prober.go:29] interesting pod/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 02 23:54:37.462417 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:37.462311 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" podUID="9fa13122-f046-4621-bf87-d124725da22f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 02 23:54:38.461135 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:38.461102 5076 patch_prober.go:29] interesting pod/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 02 23:54:38.461551 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:38.461154 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" podUID="9fa13122-f046-4621-bf87-d124725da22f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 02 23:54:39.461850 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:39.461805 5076 patch_prober.go:29] interesting pod/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 02 23:54:39.462309 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:39.461881 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" podUID="9fa13122-f046-4621-bf87-d124725da22f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 02 23:54:39.884569 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:54:39.884478 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:54:40.461431 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:40.461396 5076 patch_prober.go:29] interesting pod/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 02 23:54:40.461616 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:40.461450 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" podUID="9fa13122-f046-4621-bf87-d124725da22f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 02 23:54:41.189877 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:41.189801 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" Nov 02 23:54:41.461379 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:41.461282 5076 patch_prober.go:29] interesting pod/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 02 23:54:41.461379 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:41.461345 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" podUID="9fa13122-f046-4621-bf87-d124725da22f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 02 23:54:42.466303 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:42.466271 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" Nov 02 23:54:42.467235 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:54:42.467211 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" Nov 02 23:55:09.922795 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:55:09.922754 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:55:36.611682 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:55:36.611646 5076 generic.go:356] "Generic (PLEG): container finished" podID="56fa477d-d2c0-4a6b-a3b4-51cdf5420002" containerID="1727703d39998e9704197b736d617afd642a267537407ef2852dd65c691ee0d3" exitCode=0 Nov 02 23:55:36.612168 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:55:36.611717 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" event={"ID":"56fa477d-d2c0-4a6b-a3b4-51cdf5420002","Type":"ContainerDied","Data":"1727703d39998e9704197b736d617afd642a267537407ef2852dd65c691ee0d3"} Nov 02 23:55:36.612219 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:55:36.612167 5076 scope.go:122] "RemoveContainer" containerID="1727703d39998e9704197b736d617afd642a267537407ef2852dd65c691ee0d3" Nov 02 23:55:36.613919 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:55:36.613876 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:55:36.614037 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:55:36.613964 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:55:36.614037 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:55:36.614000 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:55:37.630539 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:55:37.630498 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" event={"ID":"56fa477d-d2c0-4a6b-a3b4-51cdf5420002","Type":"ContainerStarted","Data":"8416e217afdbe07c8c6c0566892e65f356cdf325120eb0bf69ca9170b4a2d521"} Nov 02 23:55:37.993115 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:55:37.993077 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" Nov 02 23:55:37.994162 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:55:37.994135 5076 patch_prober.go:29] interesting pod/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 02 23:55:37.994226 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:55:37.994188 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" podUID="56fa477d-d2c0-4a6b-a3b4-51cdf5420002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 02 23:55:39.010915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:55:39.010280 5076 patch_prober.go:29] interesting pod/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 02 23:55:39.010915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:55:39.010339 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" podUID="56fa477d-d2c0-4a6b-a3b4-51cdf5420002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 02 23:55:39.884395 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:55:39.884362 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:55:39.993147 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:55:39.993112 5076 patch_prober.go:29] interesting pod/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 02 23:55:39.993364 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:55:39.993182 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" podUID="56fa477d-d2c0-4a6b-a3b4-51cdf5420002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 02 23:55:40.993065 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:55:40.993027 5076 patch_prober.go:29] interesting pod/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 02 23:55:40.993456 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:55:40.993082 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" podUID="56fa477d-d2c0-4a6b-a3b4-51cdf5420002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 02 23:55:41.992943 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:55:41.992878 5076 patch_prober.go:29] interesting pod/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 02 23:55:41.993136 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:55:41.992957 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" podUID="56fa477d-d2c0-4a6b-a3b4-51cdf5420002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 02 23:55:42.655774 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:55:42.655735 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" Nov 02 23:55:42.992964 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:55:42.992930 5076 patch_prober.go:29] interesting pod/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 02 23:55:42.993146 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:55:42.992996 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" podUID="56fa477d-d2c0-4a6b-a3b4-51cdf5420002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 02 23:55:43.997735 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:55:43.997704 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" Nov 02 23:55:43.998798 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:55:43.998773 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" Nov 02 23:56:09.884561 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:56:09.884516 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:56:33.353145 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:33.353108 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw"] Nov 02 23:56:33.353737 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:33.353349 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw" podUID="519907c3-2bc2-490b-a888-ee3b91ba5f8a" containerName="maas-api" containerID="cri-o://ccdb4f79760af50d6de0c853c30256f0bc308f8741cc210b068890f8d7b5ccc0" gracePeriod=30 Nov 02 23:56:33.668519 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:33.668493 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw" Nov 02 23:56:33.695935 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:33.695880 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/519907c3-2bc2-490b-a888-ee3b91ba5f8a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519907c3-2bc2-490b-a888-ee3b91ba5f8a-openshift-service-ca\") pod \"519907c3-2bc2-490b-a888-ee3b91ba5f8a\" (UID: \"519907c3-2bc2-490b-a888-ee3b91ba5f8a\") " Nov 02 23:56:33.696092 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:33.696028 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/519907c3-2bc2-490b-a888-ee3b91ba5f8a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/519907c3-2bc2-490b-a888-ee3b91ba5f8a-metrics-certs\") pod \"519907c3-2bc2-490b-a888-ee3b91ba5f8a\" (UID: \"519907c3-2bc2-490b-a888-ee3b91ba5f8a\") " Nov 02 23:56:33.696153 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:33.696138 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/519907c3-2bc2-490b-a888-ee3b91ba5f8a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519907c3-2bc2-490b-a888-ee3b91ba5f8a-maas-api-tls\") pod \"519907c3-2bc2-490b-a888-ee3b91ba5f8a\" (UID: \"519907c3-2bc2-490b-a888-ee3b91ba5f8a\") " Nov 02 23:56:33.696302 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:33.696280 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/519907c3-2bc2-490b-a888-ee3b91ba5f8a-openshift-service-ca" pod "519907c3-2bc2-490b-a888-ee3b91ba5f8a" (UID: "519907c3-2bc2-490b-a888-ee3b91ba5f8a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:56:33.696410 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:33.696285 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/519907c3-2bc2-490b-a888-ee3b91ba5f8a-kube-api-access-x8pbx\" (UniqueName: \"kubernetes.io/projected/519907c3-2bc2-490b-a888-ee3b91ba5f8a-kube-api-access-x8pbx\") pod \"519907c3-2bc2-490b-a888-ee3b91ba5f8a\" (UID: \"519907c3-2bc2-490b-a888-ee3b91ba5f8a\") " Nov 02 23:56:33.696837 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:33.696815 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519907c3-2bc2-490b-a888-ee3b91ba5f8a-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:56:33.698087 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:33.698056 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/519907c3-2bc2-490b-a888-ee3b91ba5f8a-metrics-certs" pod "519907c3-2bc2-490b-a888-ee3b91ba5f8a" (UID: "519907c3-2bc2-490b-a888-ee3b91ba5f8a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:56:33.698183 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:33.698142 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/519907c3-2bc2-490b-a888-ee3b91ba5f8a-maas-api-tls" pod "519907c3-2bc2-490b-a888-ee3b91ba5f8a" (UID: "519907c3-2bc2-490b-a888-ee3b91ba5f8a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:56:33.698244 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:33.698228 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/519907c3-2bc2-490b-a888-ee3b91ba5f8a-kube-api-access-x8pbx" pod "519907c3-2bc2-490b-a888-ee3b91ba5f8a" (UID: "519907c3-2bc2-490b-a888-ee3b91ba5f8a"). InnerVolumeSpecName "kube-api-access-x8pbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:56:33.797948 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:33.797911 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/519907c3-2bc2-490b-a888-ee3b91ba5f8a-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:56:33.797948 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:33.797949 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519907c3-2bc2-490b-a888-ee3b91ba5f8a-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:56:33.798173 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:33.797967 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8pbx\" (UniqueName: \"kubernetes.io/projected/519907c3-2bc2-490b-a888-ee3b91ba5f8a-kube-api-access-x8pbx\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:56:34.012683 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:34.012643 5076 generic.go:356] "Generic (PLEG): container finished" podID="519907c3-2bc2-490b-a888-ee3b91ba5f8a" containerID="ccdb4f79760af50d6de0c853c30256f0bc308f8741cc210b068890f8d7b5ccc0" exitCode=0 Nov 02 23:56:34.012879 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:34.012694 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw" Nov 02 23:56:34.012879 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:34.012724 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw" event={"ID":"519907c3-2bc2-490b-a888-ee3b91ba5f8a","Type":"ContainerDied","Data":"ccdb4f79760af50d6de0c853c30256f0bc308f8741cc210b068890f8d7b5ccc0"} Nov 02 23:56:34.012879 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:34.012783 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw" event={"ID":"519907c3-2bc2-490b-a888-ee3b91ba5f8a","Type":"ContainerDied","Data":"24ed1732fb0cdf265c76371dd563100ab4fbc747dedc4a025b7fec0acffebdd8"} Nov 02 23:56:34.012879 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:34.012815 5076 scope.go:122] "RemoveContainer" containerID="ccdb4f79760af50d6de0c853c30256f0bc308f8741cc210b068890f8d7b5ccc0" Nov 02 23:56:34.023557 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:34.023537 5076 scope.go:122] "RemoveContainer" containerID="ccdb4f79760af50d6de0c853c30256f0bc308f8741cc210b068890f8d7b5ccc0" Nov 02 23:56:34.023851 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:56:34.023829 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccdb4f79760af50d6de0c853c30256f0bc308f8741cc210b068890f8d7b5ccc0\": container with ID starting with ccdb4f79760af50d6de0c853c30256f0bc308f8741cc210b068890f8d7b5ccc0 not found: ID does not exist" containerID="ccdb4f79760af50d6de0c853c30256f0bc308f8741cc210b068890f8d7b5ccc0" Nov 02 23:56:34.023919 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:34.023860 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccdb4f79760af50d6de0c853c30256f0bc308f8741cc210b068890f8d7b5ccc0"} err="failed to get container status \"ccdb4f79760af50d6de0c853c30256f0bc308f8741cc210b068890f8d7b5ccc0\": rpc error: code = NotFound desc = could not find container \"ccdb4f79760af50d6de0c853c30256f0bc308f8741cc210b068890f8d7b5ccc0\": container with ID starting with ccdb4f79760af50d6de0c853c30256f0bc308f8741cc210b068890f8d7b5ccc0 not found: ID does not exist" Nov 02 23:56:34.171794 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:34.171749 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw"] Nov 02 23:56:34.177530 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:34.177499 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3bcb9f-5496655ff4-p6rvw"] Nov 02 23:56:35.168278 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:35.168244 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="519907c3-2bc2-490b-a888-ee3b91ba5f8a" path="/var/lib/kubelet/pods/519907c3-2bc2-490b-a888-ee3b91ba5f8a/volumes" Nov 02 23:56:39.896993 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:56:39.896957 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:56:39.920093 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:39.920064 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x"] Nov 02 23:56:39.920294 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:39.920274 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" podUID="d2656fde-164f-40ec-845f-ad9c08091597" containerName="istio-proxy" containerID="cri-o://3cd4f87e29594cb2d3c3c442e9b8b343254094128c5552416704cee25dc14979" gracePeriod=30 Nov 02 23:56:45.167394 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:45.167361 5076 generic.go:356] "Generic (PLEG): container finished" podID="d2656fde-164f-40ec-845f-ad9c08091597" containerID="3cd4f87e29594cb2d3c3c442e9b8b343254094128c5552416704cee25dc14979" exitCode=0 Nov 02 23:56:45.167924 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:45.167433 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" event={"ID":"d2656fde-164f-40ec-845f-ad9c08091597","Type":"ContainerDied","Data":"3cd4f87e29594cb2d3c3c442e9b8b343254094128c5552416704cee25dc14979"} Nov 02 23:56:45.167924 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:45.167507 5076 scope.go:122] "RemoveContainer" containerID="4761c7f31292c82638a9e7003c366e65f2dfa51e398e2b9631698ea65367f329" Nov 02 23:56:45.206537 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:45.206515 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" Nov 02 23:56:45.392401 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:45.392364 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2656fde-164f-40ec-845f-ad9c08091597-kube-api-access-z4rnl\" (UniqueName: \"kubernetes.io/projected/d2656fde-164f-40ec-845f-ad9c08091597-kube-api-access-z4rnl\") pod \"d2656fde-164f-40ec-845f-ad9c08091597\" (UID: \"d2656fde-164f-40ec-845f-ad9c08091597\") " Nov 02 23:56:45.392401 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:45.392405 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2656fde-164f-40ec-845f-ad9c08091597-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2656fde-164f-40ec-845f-ad9c08091597-istio-envoy\") pod \"d2656fde-164f-40ec-845f-ad9c08091597\" (UID: \"d2656fde-164f-40ec-845f-ad9c08091597\") " Nov 02 23:56:45.392666 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:45.392457 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2656fde-164f-40ec-845f-ad9c08091597-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2656fde-164f-40ec-845f-ad9c08091597-istio-podinfo\") pod \"d2656fde-164f-40ec-845f-ad9c08091597\" (UID: \"d2656fde-164f-40ec-845f-ad9c08091597\") " Nov 02 23:56:45.392666 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:45.392515 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2656fde-164f-40ec-845f-ad9c08091597-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2656fde-164f-40ec-845f-ad9c08091597-istio-data\") pod \"d2656fde-164f-40ec-845f-ad9c08091597\" (UID: \"d2656fde-164f-40ec-845f-ad9c08091597\") " Nov 02 23:56:45.392666 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:45.392613 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2656fde-164f-40ec-845f-ad9c08091597-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2656fde-164f-40ec-845f-ad9c08091597-workload-socket\") pod \"d2656fde-164f-40ec-845f-ad9c08091597\" (UID: \"d2656fde-164f-40ec-845f-ad9c08091597\") " Nov 02 23:56:45.392970 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:45.392945 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2656fde-164f-40ec-845f-ad9c08091597-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2656fde-164f-40ec-845f-ad9c08091597-workload-certs\") pod \"d2656fde-164f-40ec-845f-ad9c08091597\" (UID: \"d2656fde-164f-40ec-845f-ad9c08091597\") " Nov 02 23:56:45.393086 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:45.392965 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2656fde-164f-40ec-845f-ad9c08091597-istio-data" pod "d2656fde-164f-40ec-845f-ad9c08091597" (UID: "d2656fde-164f-40ec-845f-ad9c08091597"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:56:45.393086 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:45.392996 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2656fde-164f-40ec-845f-ad9c08091597-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2656fde-164f-40ec-845f-ad9c08091597-credential-socket\") pod \"d2656fde-164f-40ec-845f-ad9c08091597\" (UID: \"d2656fde-164f-40ec-845f-ad9c08091597\") " Nov 02 23:56:45.393198 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:45.393108 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2656fde-164f-40ec-845f-ad9c08091597-istio-token\" (UniqueName: \"kubernetes.io/projected/d2656fde-164f-40ec-845f-ad9c08091597-istio-token\") pod \"d2656fde-164f-40ec-845f-ad9c08091597\" (UID: \"d2656fde-164f-40ec-845f-ad9c08091597\") " Nov 02 23:56:45.393198 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:45.393188 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2656fde-164f-40ec-845f-ad9c08091597-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2656fde-164f-40ec-845f-ad9c08091597-istiod-ca-cert\") pod \"d2656fde-164f-40ec-845f-ad9c08091597\" (UID: \"d2656fde-164f-40ec-845f-ad9c08091597\") " Nov 02 23:56:45.393574 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:45.393545 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2656fde-164f-40ec-845f-ad9c08091597-workload-certs" pod "d2656fde-164f-40ec-845f-ad9c08091597" (UID: "d2656fde-164f-40ec-845f-ad9c08091597"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:56:45.393724 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:45.393689 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2656fde-164f-40ec-845f-ad9c08091597-workload-socket" pod "d2656fde-164f-40ec-845f-ad9c08091597" (UID: "d2656fde-164f-40ec-845f-ad9c08091597"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:56:45.393724 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:45.393689 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2656fde-164f-40ec-845f-ad9c08091597-credential-socket" pod "d2656fde-164f-40ec-845f-ad9c08091597" (UID: "d2656fde-164f-40ec-845f-ad9c08091597"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:56:45.393851 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:45.393808 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2656fde-164f-40ec-845f-ad9c08091597-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:56:45.393851 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:45.393825 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2656fde-164f-40ec-845f-ad9c08091597-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:56:45.393851 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:45.393836 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2656fde-164f-40ec-845f-ad9c08091597-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:56:45.393851 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:45.393844 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2656fde-164f-40ec-845f-ad9c08091597-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:56:45.394187 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:45.394159 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2656fde-164f-40ec-845f-ad9c08091597-istiod-ca-cert" pod "d2656fde-164f-40ec-845f-ad9c08091597" (UID: "d2656fde-164f-40ec-845f-ad9c08091597"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:56:45.395365 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:45.395335 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2656fde-164f-40ec-845f-ad9c08091597-kube-api-access-z4rnl" pod "d2656fde-164f-40ec-845f-ad9c08091597" (UID: "d2656fde-164f-40ec-845f-ad9c08091597"). InnerVolumeSpecName "kube-api-access-z4rnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:56:45.395753 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:45.395731 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2656fde-164f-40ec-845f-ad9c08091597-istio-envoy" pod "d2656fde-164f-40ec-845f-ad9c08091597" (UID: "d2656fde-164f-40ec-845f-ad9c08091597"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:56:45.395859 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:45.395845 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2656fde-164f-40ec-845f-ad9c08091597-istio-podinfo" pod "d2656fde-164f-40ec-845f-ad9c08091597" (UID: "d2656fde-164f-40ec-845f-ad9c08091597"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:56:45.396739 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:45.396712 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2656fde-164f-40ec-845f-ad9c08091597-istio-token" pod "d2656fde-164f-40ec-845f-ad9c08091597" (UID: "d2656fde-164f-40ec-845f-ad9c08091597"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:56:45.495245 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:45.495148 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2656fde-164f-40ec-845f-ad9c08091597-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:56:45.495245 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:45.495184 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4rnl\" (UniqueName: \"kubernetes.io/projected/d2656fde-164f-40ec-845f-ad9c08091597-kube-api-access-z4rnl\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:56:45.495245 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:45.495199 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2656fde-164f-40ec-845f-ad9c08091597-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:56:45.495245 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:45.495215 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2656fde-164f-40ec-845f-ad9c08091597-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:56:45.495245 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:45.495226 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2656fde-164f-40ec-845f-ad9c08091597-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:56:46.175824 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:46.175795 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" Nov 02 23:56:46.176274 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:46.175833 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x" event={"ID":"d2656fde-164f-40ec-845f-ad9c08091597","Type":"ContainerDied","Data":"abf60cb25aae88dcc129f025ec8c810db2848e409d63a34c80b413ed80ed11a3"} Nov 02 23:56:46.176274 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:46.175885 5076 scope.go:122] "RemoveContainer" containerID="3cd4f87e29594cb2d3c3c442e9b8b343254094128c5552416704cee25dc14979" Nov 02 23:56:46.379780 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:46.379741 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x"] Nov 02 23:56:46.427307 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:46.427228 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-3bcb9f-openshift-default-66d8d4885c-66m7x"] Nov 02 23:56:47.148992 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:47.148951 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2656fde-164f-40ec-845f-ad9c08091597" path="/var/lib/kubelet/pods/d2656fde-164f-40ec-845f-ad9c08091597/volumes" Nov 02 23:56:55.068974 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:55.068938 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t"] Nov 02 23:56:55.069404 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:55.069169 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t" podUID="c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa" containerName="maas-api" containerID="cri-o://0e3d3fb469e8fa4e94859c2802244196ae83eb92249761c7b9903881cd6d00a4" gracePeriod=30 Nov 02 23:56:55.237113 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:55.237082 5076 generic.go:356] "Generic (PLEG): container finished" podID="c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa" containerID="0e3d3fb469e8fa4e94859c2802244196ae83eb92249761c7b9903881cd6d00a4" exitCode=0 Nov 02 23:56:55.237301 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:55.237158 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t" event={"ID":"c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa","Type":"ContainerDied","Data":"0e3d3fb469e8fa4e94859c2802244196ae83eb92249761c7b9903881cd6d00a4"} Nov 02 23:56:55.385539 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:55.385513 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t" Nov 02 23:56:55.433163 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:55.433129 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa-maas-api-tls\") pod \"c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa\" (UID: \"c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa\") " Nov 02 23:56:55.433361 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:55.433193 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa-kube-api-access-fbkwk\" (UniqueName: \"kubernetes.io/projected/c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa-kube-api-access-fbkwk\") pod \"c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa\" (UID: \"c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa\") " Nov 02 23:56:55.433361 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:55.433290 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa-openshift-service-ca\") pod \"c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa\" (UID: \"c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa\") " Nov 02 23:56:55.433361 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:55.433332 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa-metrics-certs\") pod \"c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa\" (UID: \"c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa\") " Nov 02 23:56:55.433725 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:55.433692 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa-openshift-service-ca" pod "c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa" (UID: "c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:56:55.433885 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:55.433868 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:56:55.435317 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:55.435297 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa-maas-api-tls" pod "c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa" (UID: "c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:56:55.435422 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:55.435379 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa-kube-api-access-fbkwk" pod "c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa" (UID: "c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa"). InnerVolumeSpecName "kube-api-access-fbkwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:56:55.435422 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:55.435414 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa-metrics-certs" pod "c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa" (UID: "c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:56:55.535852 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:55.535812 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:56:55.535852 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:55.535853 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbkwk\" (UniqueName: \"kubernetes.io/projected/c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa-kube-api-access-fbkwk\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:56:55.536134 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:55.535870 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:56:56.245315 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:56.245283 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t" Nov 02 23:56:56.245761 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:56.245285 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t" event={"ID":"c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa","Type":"ContainerDied","Data":"88f0ad75e4f6a9f295b1a376aed6021866d78d5b8bd5a09dee46d0f0db4b1f9c"} Nov 02 23:56:56.245761 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:56.245417 5076 scope.go:122] "RemoveContainer" containerID="0e3d3fb469e8fa4e94859c2802244196ae83eb92249761c7b9903881cd6d00a4" Nov 02 23:56:56.402937 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:56.402903 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t"] Nov 02 23:56:56.406644 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:56.406615 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c2bff-647db7bb88-b788t"] Nov 02 23:56:57.148356 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:57.148309 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa" path="/var/lib/kubelet/pods/c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa/volumes" Nov 02 23:56:59.110273 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:59.110240 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n"] Nov 02 23:56:59.110678 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:56:59.110453 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" podUID="7947bd91-9f2d-44dc-9e12-d3c8f3744ab0" containerName="istio-proxy" containerID="cri-o://7837c3ff776b93b2944c856efbbae2b2d6af94b6ecf8c5f8db0dbe2664422f44" gracePeriod=30 Nov 02 23:57:04.328163 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:04.328123 5076 generic.go:356] "Generic (PLEG): container finished" podID="7947bd91-9f2d-44dc-9e12-d3c8f3744ab0" containerID="7837c3ff776b93b2944c856efbbae2b2d6af94b6ecf8c5f8db0dbe2664422f44" exitCode=0 Nov 02 23:57:04.328631 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:04.328195 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" event={"ID":"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0","Type":"ContainerDied","Data":"7837c3ff776b93b2944c856efbbae2b2d6af94b6ecf8c5f8db0dbe2664422f44"} Nov 02 23:57:04.328631 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:04.328275 5076 scope.go:122] "RemoveContainer" containerID="e7a104e2d6299ccea44e62b5db27a57544f56dc0d2828eb158201b72a8c2a40d" Nov 02 23:57:04.384108 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:04.384042 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" Nov 02 23:57:04.458103 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:04.458069 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-istio-envoy\") pod \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\" (UID: \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\") " Nov 02 23:57:04.458103 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:04.458109 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-istio-token\" (UniqueName: \"kubernetes.io/projected/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-istio-token\") pod \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\" (UID: \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\") " Nov 02 23:57:04.458311 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:04.458224 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-workload-socket\") pod \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\" (UID: \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\") " Nov 02 23:57:04.458311 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:04.458249 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-workload-certs\") pod \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\" (UID: \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\") " Nov 02 23:57:04.458311 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:04.458281 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-istiod-ca-cert\") pod \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\" (UID: \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\") " Nov 02 23:57:04.458446 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:04.458339 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-credential-socket\") pod \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\" (UID: \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\") " Nov 02 23:57:04.458446 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:04.458367 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-istio-data\") pod \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\" (UID: \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\") " Nov 02 23:57:04.458446 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:04.458418 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-kube-api-access-ggl8t\" (UniqueName: \"kubernetes.io/projected/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-kube-api-access-ggl8t\") pod \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\" (UID: \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\") " Nov 02 23:57:04.458611 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:04.458457 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-istio-podinfo\") pod \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\" (UID: \"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0\") " Nov 02 23:57:04.458611 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:04.458573 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-workload-certs" pod "7947bd91-9f2d-44dc-9e12-d3c8f3744ab0" (UID: "7947bd91-9f2d-44dc-9e12-d3c8f3744ab0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:57:04.458611 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:04.458583 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-workload-socket" pod "7947bd91-9f2d-44dc-9e12-d3c8f3744ab0" (UID: "7947bd91-9f2d-44dc-9e12-d3c8f3744ab0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:57:04.458798 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:04.458660 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-credential-socket" pod "7947bd91-9f2d-44dc-9e12-d3c8f3744ab0" (UID: "7947bd91-9f2d-44dc-9e12-d3c8f3744ab0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:57:04.458798 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:04.458780 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-istio-data" pod "7947bd91-9f2d-44dc-9e12-d3c8f3744ab0" (UID: "7947bd91-9f2d-44dc-9e12-d3c8f3744ab0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:57:04.458798 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:04.458780 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-istiod-ca-cert" pod "7947bd91-9f2d-44dc-9e12-d3c8f3744ab0" (UID: "7947bd91-9f2d-44dc-9e12-d3c8f3744ab0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:57:04.458979 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:04.458961 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:04.458979 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:04.458975 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:04.459058 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:04.458985 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:04.459058 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:04.458994 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:04.459058 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:04.459002 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:04.460800 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:04.460770 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-istio-podinfo" pod "7947bd91-9f2d-44dc-9e12-d3c8f3744ab0" (UID: "7947bd91-9f2d-44dc-9e12-d3c8f3744ab0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:57:04.461006 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:04.460799 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-kube-api-access-ggl8t" pod "7947bd91-9f2d-44dc-9e12-d3c8f3744ab0" (UID: "7947bd91-9f2d-44dc-9e12-d3c8f3744ab0"). InnerVolumeSpecName "kube-api-access-ggl8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:57:04.461006 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:04.460867 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-istio-envoy" pod "7947bd91-9f2d-44dc-9e12-d3c8f3744ab0" (UID: "7947bd91-9f2d-44dc-9e12-d3c8f3744ab0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:57:04.461006 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:04.460882 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-istio-token" pod "7947bd91-9f2d-44dc-9e12-d3c8f3744ab0" (UID: "7947bd91-9f2d-44dc-9e12-d3c8f3744ab0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:57:04.559882 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:04.559851 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggl8t\" (UniqueName: \"kubernetes.io/projected/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-kube-api-access-ggl8t\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:04.559882 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:04.559879 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:04.559882 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:04.559905 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:04.560138 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:04.559914 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:05.335153 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:05.335119 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" Nov 02 23:57:05.335153 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:05.335136 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n" event={"ID":"7947bd91-9f2d-44dc-9e12-d3c8f3744ab0","Type":"ContainerDied","Data":"1967e77cd5fb50f35607ce483c912a927775a7edf98b629e1327fa37c81811e4"} Nov 02 23:57:05.335713 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:05.335213 5076 scope.go:122] "RemoveContainer" containerID="7837c3ff776b93b2944c856efbbae2b2d6af94b6ecf8c5f8db0dbe2664422f44" Nov 02 23:57:05.542639 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:05.542597 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n"] Nov 02 23:57:05.590182 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:05.590148 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-8c2bff-openshift-default-6545bcd57c-lsr2n"] Nov 02 23:57:06.121054 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:06.121016 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr"] Nov 02 23:57:06.121267 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:06.121243 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr" podUID="22a0ed3d-b728-4e61-81a2-1c118261632c" containerName="maas-api" containerID="cri-o://5d62c65bd3ca176e5e9dca96927ee6be92018b8167832742f05365cce1e751ce" gracePeriod=30 Nov 02 23:57:06.403976 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:06.403554 5076 generic.go:356] "Generic (PLEG): container finished" podID="22a0ed3d-b728-4e61-81a2-1c118261632c" containerID="5d62c65bd3ca176e5e9dca96927ee6be92018b8167832742f05365cce1e751ce" exitCode=0 Nov 02 23:57:06.403976 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:06.403638 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr" event={"ID":"22a0ed3d-b728-4e61-81a2-1c118261632c","Type":"ContainerDied","Data":"5d62c65bd3ca176e5e9dca96927ee6be92018b8167832742f05365cce1e751ce"} Nov 02 23:57:06.444305 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:06.444279 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr" Nov 02 23:57:06.488552 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:06.488513 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22a0ed3d-b728-4e61-81a2-1c118261632c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a0ed3d-b728-4e61-81a2-1c118261632c-openshift-service-ca\") pod \"22a0ed3d-b728-4e61-81a2-1c118261632c\" (UID: \"22a0ed3d-b728-4e61-81a2-1c118261632c\") " Nov 02 23:57:06.488747 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:06.488591 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22a0ed3d-b728-4e61-81a2-1c118261632c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22a0ed3d-b728-4e61-81a2-1c118261632c-maas-api-tls\") pod \"22a0ed3d-b728-4e61-81a2-1c118261632c\" (UID: \"22a0ed3d-b728-4e61-81a2-1c118261632c\") " Nov 02 23:57:06.488747 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:06.488695 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22a0ed3d-b728-4e61-81a2-1c118261632c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/22a0ed3d-b728-4e61-81a2-1c118261632c-metrics-certs\") pod \"22a0ed3d-b728-4e61-81a2-1c118261632c\" (UID: \"22a0ed3d-b728-4e61-81a2-1c118261632c\") " Nov 02 23:57:06.488960 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:06.488931 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22a0ed3d-b728-4e61-81a2-1c118261632c-openshift-service-ca" pod "22a0ed3d-b728-4e61-81a2-1c118261632c" (UID: "22a0ed3d-b728-4e61-81a2-1c118261632c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:57:06.489031 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:06.489001 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22a0ed3d-b728-4e61-81a2-1c118261632c-kube-api-access-mzcvh\" (UniqueName: \"kubernetes.io/projected/22a0ed3d-b728-4e61-81a2-1c118261632c-kube-api-access-mzcvh\") pod \"22a0ed3d-b728-4e61-81a2-1c118261632c\" (UID: \"22a0ed3d-b728-4e61-81a2-1c118261632c\") " Nov 02 23:57:06.489436 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:06.489416 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a0ed3d-b728-4e61-81a2-1c118261632c-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:06.490785 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:06.490756 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22a0ed3d-b728-4e61-81a2-1c118261632c-maas-api-tls" pod "22a0ed3d-b728-4e61-81a2-1c118261632c" (UID: "22a0ed3d-b728-4e61-81a2-1c118261632c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:57:06.490785 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:06.490775 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22a0ed3d-b728-4e61-81a2-1c118261632c-metrics-certs" pod "22a0ed3d-b728-4e61-81a2-1c118261632c" (UID: "22a0ed3d-b728-4e61-81a2-1c118261632c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:57:06.490951 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:06.490870 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22a0ed3d-b728-4e61-81a2-1c118261632c-kube-api-access-mzcvh" pod "22a0ed3d-b728-4e61-81a2-1c118261632c" (UID: "22a0ed3d-b728-4e61-81a2-1c118261632c"). InnerVolumeSpecName "kube-api-access-mzcvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:57:06.590266 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:06.590224 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzcvh\" (UniqueName: \"kubernetes.io/projected/22a0ed3d-b728-4e61-81a2-1c118261632c-kube-api-access-mzcvh\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:06.590266 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:06.590264 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22a0ed3d-b728-4e61-81a2-1c118261632c-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:06.590473 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:06.590278 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22a0ed3d-b728-4e61-81a2-1c118261632c-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:07.181386 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:07.181251 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7947bd91-9f2d-44dc-9e12-d3c8f3744ab0" path="/var/lib/kubelet/pods/7947bd91-9f2d-44dc-9e12-d3c8f3744ab0/volumes" Nov 02 23:57:07.359911 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:07.359800 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7"] Nov 02 23:57:07.360127 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:07.360053 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7" podUID="73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b" containerName="maas-api" containerID="cri-o://a47577f1e5abf31903a850e979cf2f637c0b1a71294d3d7f0376344c9f79f99e" gracePeriod=30 Nov 02 23:57:07.473850 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:07.473814 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr" event={"ID":"22a0ed3d-b728-4e61-81a2-1c118261632c","Type":"ContainerDied","Data":"d5464b63232fdce8c4d876527147d763fdd96b700b1ec15966623b9f06b4e865"} Nov 02 23:57:07.474375 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:07.473876 5076 scope.go:122] "RemoveContainer" containerID="5d62c65bd3ca176e5e9dca96927ee6be92018b8167832742f05365cce1e751ce" Nov 02 23:57:07.474375 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:07.474062 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr" Nov 02 23:57:07.636204 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:07.636174 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr"] Nov 02 23:57:07.638792 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:07.638768 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4b63d8-6cc7bffc4c-8vshr"] Nov 02 23:57:07.732687 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:07.732321 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7" Nov 02 23:57:07.819537 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:07.819497 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b-openshift-service-ca\") pod \"73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b\" (UID: \"73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b\") " Nov 02 23:57:07.819698 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:07.819592 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b-kube-api-access-pzbk4\" (UniqueName: \"kubernetes.io/projected/73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b-kube-api-access-pzbk4\") pod \"73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b\" (UID: \"73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b\") " Nov 02 23:57:07.819745 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:07.819706 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b-metrics-certs\") pod \"73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b\" (UID: \"73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b\") " Nov 02 23:57:07.819819 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:07.819749 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b-maas-api-tls\") pod \"73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b\" (UID: \"73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b\") " Nov 02 23:57:07.819925 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:07.819858 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b-openshift-service-ca" pod "73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b" (UID: "73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:57:07.820166 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:07.820137 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:07.821951 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:07.821919 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b-metrics-certs" pod "73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b" (UID: "73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:57:07.822074 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:07.821922 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b-kube-api-access-pzbk4" pod "73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b" (UID: "73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b"). InnerVolumeSpecName "kube-api-access-pzbk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:57:07.822074 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:07.821991 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b-maas-api-tls" pod "73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b" (UID: "73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:57:07.921217 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:07.921183 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:07.921217 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:07.921211 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:07.921217 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:07.921220 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzbk4\" (UniqueName: \"kubernetes.io/projected/73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b-kube-api-access-pzbk4\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:08.483118 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:08.483081 5076 generic.go:356] "Generic (PLEG): container finished" podID="73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b" containerID="a47577f1e5abf31903a850e979cf2f637c0b1a71294d3d7f0376344c9f79f99e" exitCode=0 Nov 02 23:57:08.483568 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:08.483131 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7" Nov 02 23:57:08.483568 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:08.483144 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7" event={"ID":"73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b","Type":"ContainerDied","Data":"a47577f1e5abf31903a850e979cf2f637c0b1a71294d3d7f0376344c9f79f99e"} Nov 02 23:57:08.483568 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:08.483197 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7" event={"ID":"73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b","Type":"ContainerDied","Data":"efb23a83712cbf4678fddd0c5a48fae6c89595a65c4dc715631da3a3eaa334f5"} Nov 02 23:57:08.483568 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:08.483227 5076 scope.go:122] "RemoveContainer" containerID="a47577f1e5abf31903a850e979cf2f637c0b1a71294d3d7f0376344c9f79f99e" Nov 02 23:57:08.493571 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:08.493550 5076 scope.go:122] "RemoveContainer" containerID="a47577f1e5abf31903a850e979cf2f637c0b1a71294d3d7f0376344c9f79f99e" Nov 02 23:57:08.493843 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:57:08.493822 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a47577f1e5abf31903a850e979cf2f637c0b1a71294d3d7f0376344c9f79f99e\": container with ID starting with a47577f1e5abf31903a850e979cf2f637c0b1a71294d3d7f0376344c9f79f99e not found: ID does not exist" containerID="a47577f1e5abf31903a850e979cf2f637c0b1a71294d3d7f0376344c9f79f99e" Nov 02 23:57:08.493914 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:08.493854 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a47577f1e5abf31903a850e979cf2f637c0b1a71294d3d7f0376344c9f79f99e"} err="failed to get container status \"a47577f1e5abf31903a850e979cf2f637c0b1a71294d3d7f0376344c9f79f99e\": rpc error: code = NotFound desc = could not find container \"a47577f1e5abf31903a850e979cf2f637c0b1a71294d3d7f0376344c9f79f99e\": container with ID starting with a47577f1e5abf31903a850e979cf2f637c0b1a71294d3d7f0376344c9f79f99e not found: ID does not exist" Nov 02 23:57:08.640155 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:08.640118 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7"] Nov 02 23:57:08.687371 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:08.687335 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f2df33-65744469d5-nczc7"] Nov 02 23:57:09.147877 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:09.147839 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22a0ed3d-b728-4e61-81a2-1c118261632c" path="/var/lib/kubelet/pods/22a0ed3d-b728-4e61-81a2-1c118261632c/volumes" Nov 02 23:57:09.148321 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:09.148306 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b" path="/var/lib/kubelet/pods/73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b/volumes" Nov 02 23:57:09.885315 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:57:09.885282 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:57:13.530108 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:13.530069 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2"] Nov 02 23:57:13.530593 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:13.530351 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" podUID="9fa13122-f046-4621-bf87-d124725da22f" containerName="istio-proxy" containerID="cri-o://eb588c2e1e7d6a7f92ea7a9fea5d07fedd5840c8b3365b664c3d701f1b325b65" gracePeriod=30 Nov 02 23:57:18.799544 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:18.799473 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" Nov 02 23:57:18.831761 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:18.831727 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fa13122-f046-4621-bf87-d124725da22f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fa13122-f046-4621-bf87-d124725da22f-istio-podinfo\") pod \"9fa13122-f046-4621-bf87-d124725da22f\" (UID: \"9fa13122-f046-4621-bf87-d124725da22f\") " Nov 02 23:57:18.831950 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:18.831775 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fa13122-f046-4621-bf87-d124725da22f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fa13122-f046-4621-bf87-d124725da22f-workload-socket\") pod \"9fa13122-f046-4621-bf87-d124725da22f\" (UID: \"9fa13122-f046-4621-bf87-d124725da22f\") " Nov 02 23:57:18.831950 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:18.831811 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fa13122-f046-4621-bf87-d124725da22f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fa13122-f046-4621-bf87-d124725da22f-workload-certs\") pod \"9fa13122-f046-4621-bf87-d124725da22f\" (UID: \"9fa13122-f046-4621-bf87-d124725da22f\") " Nov 02 23:57:18.831950 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:18.831846 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fa13122-f046-4621-bf87-d124725da22f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fa13122-f046-4621-bf87-d124725da22f-istio-envoy\") pod \"9fa13122-f046-4621-bf87-d124725da22f\" (UID: \"9fa13122-f046-4621-bf87-d124725da22f\") " Nov 02 23:57:18.831950 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:18.831924 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fa13122-f046-4621-bf87-d124725da22f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fa13122-f046-4621-bf87-d124725da22f-credential-socket\") pod \"9fa13122-f046-4621-bf87-d124725da22f\" (UID: \"9fa13122-f046-4621-bf87-d124725da22f\") " Nov 02 23:57:18.832198 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:18.831959 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fa13122-f046-4621-bf87-d124725da22f-kube-api-access-wj77t\" (UniqueName: \"kubernetes.io/projected/9fa13122-f046-4621-bf87-d124725da22f-kube-api-access-wj77t\") pod \"9fa13122-f046-4621-bf87-d124725da22f\" (UID: \"9fa13122-f046-4621-bf87-d124725da22f\") " Nov 02 23:57:18.832198 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:18.832105 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fa13122-f046-4621-bf87-d124725da22f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fa13122-f046-4621-bf87-d124725da22f-istiod-ca-cert\") pod \"9fa13122-f046-4621-bf87-d124725da22f\" (UID: \"9fa13122-f046-4621-bf87-d124725da22f\") " Nov 02 23:57:18.832198 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:18.832134 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fa13122-f046-4621-bf87-d124725da22f-istio-token\" (UniqueName: \"kubernetes.io/projected/9fa13122-f046-4621-bf87-d124725da22f-istio-token\") pod \"9fa13122-f046-4621-bf87-d124725da22f\" (UID: \"9fa13122-f046-4621-bf87-d124725da22f\") " Nov 02 23:57:18.832369 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:18.832281 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fa13122-f046-4621-bf87-d124725da22f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fa13122-f046-4621-bf87-d124725da22f-istio-data\") pod \"9fa13122-f046-4621-bf87-d124725da22f\" (UID: \"9fa13122-f046-4621-bf87-d124725da22f\") " Nov 02 23:57:18.832942 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:18.832916 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fa13122-f046-4621-bf87-d124725da22f-istio-data" pod "9fa13122-f046-4621-bf87-d124725da22f" (UID: "9fa13122-f046-4621-bf87-d124725da22f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:57:18.833115 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:18.833097 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fa13122-f046-4621-bf87-d124725da22f-credential-socket" pod "9fa13122-f046-4621-bf87-d124725da22f" (UID: "9fa13122-f046-4621-bf87-d124725da22f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:57:18.834626 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:18.834601 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fa13122-f046-4621-bf87-d124725da22f-istio-podinfo" pod "9fa13122-f046-4621-bf87-d124725da22f" (UID: "9fa13122-f046-4621-bf87-d124725da22f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:57:18.834986 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:18.834965 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fa13122-f046-4621-bf87-d124725da22f-istiod-ca-cert" pod "9fa13122-f046-4621-bf87-d124725da22f" (UID: "9fa13122-f046-4621-bf87-d124725da22f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:57:18.836980 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:18.835864 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fa13122-f046-4621-bf87-d124725da22f-workload-certs" pod "9fa13122-f046-4621-bf87-d124725da22f" (UID: "9fa13122-f046-4621-bf87-d124725da22f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:57:18.836980 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:18.836058 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fa13122-f046-4621-bf87-d124725da22f-kube-api-access-wj77t" pod "9fa13122-f046-4621-bf87-d124725da22f" (UID: "9fa13122-f046-4621-bf87-d124725da22f"). InnerVolumeSpecName "kube-api-access-wj77t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:57:18.836980 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:18.836691 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fa13122-f046-4621-bf87-d124725da22f-workload-socket" pod "9fa13122-f046-4621-bf87-d124725da22f" (UID: "9fa13122-f046-4621-bf87-d124725da22f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:57:18.838922 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:18.838863 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fa13122-f046-4621-bf87-d124725da22f-istio-token" pod "9fa13122-f046-4621-bf87-d124725da22f" (UID: "9fa13122-f046-4621-bf87-d124725da22f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:57:18.839315 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:18.839294 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fa13122-f046-4621-bf87-d124725da22f-istio-envoy" pod "9fa13122-f046-4621-bf87-d124725da22f" (UID: "9fa13122-f046-4621-bf87-d124725da22f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:57:18.934058 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:18.934020 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj77t\" (UniqueName: \"kubernetes.io/projected/9fa13122-f046-4621-bf87-d124725da22f-kube-api-access-wj77t\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:18.934058 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:18.934050 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fa13122-f046-4621-bf87-d124725da22f-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:18.934058 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:18.934059 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fa13122-f046-4621-bf87-d124725da22f-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:18.934302 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:18.934070 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fa13122-f046-4621-bf87-d124725da22f-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:18.934302 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:18.934082 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fa13122-f046-4621-bf87-d124725da22f-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:18.934302 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:18.934090 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fa13122-f046-4621-bf87-d124725da22f-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:18.934302 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:18.934099 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fa13122-f046-4621-bf87-d124725da22f-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:18.934302 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:18.934106 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fa13122-f046-4621-bf87-d124725da22f-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:18.934302 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:18.934114 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fa13122-f046-4621-bf87-d124725da22f-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:19.568608 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:19.568573 5076 generic.go:356] "Generic (PLEG): container finished" podID="9fa13122-f046-4621-bf87-d124725da22f" containerID="eb588c2e1e7d6a7f92ea7a9fea5d07fedd5840c8b3365b664c3d701f1b325b65" exitCode=0 Nov 02 23:57:19.568794 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:19.568616 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" Nov 02 23:57:19.568794 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:19.568657 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" event={"ID":"9fa13122-f046-4621-bf87-d124725da22f","Type":"ContainerDied","Data":"eb588c2e1e7d6a7f92ea7a9fea5d07fedd5840c8b3365b664c3d701f1b325b65"} Nov 02 23:57:19.568794 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:19.568715 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2" event={"ID":"9fa13122-f046-4621-bf87-d124725da22f","Type":"ContainerDied","Data":"2b35f131bd152261edf77dbd9e3c0023b7bd0c27a8c72c8bd93b08ab17ecf034"} Nov 02 23:57:19.568794 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:19.568741 5076 scope.go:122] "RemoveContainer" containerID="eb588c2e1e7d6a7f92ea7a9fea5d07fedd5840c8b3365b664c3d701f1b325b65" Nov 02 23:57:19.581394 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:19.581373 5076 scope.go:122] "RemoveContainer" containerID="09fe8c125532ecb19785795e189dfc0e452a67429777d199981bd9c26eb2e8cb" Nov 02 23:57:19.591022 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:19.591003 5076 scope.go:122] "RemoveContainer" containerID="eb588c2e1e7d6a7f92ea7a9fea5d07fedd5840c8b3365b664c3d701f1b325b65" Nov 02 23:57:19.591293 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:57:19.591269 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb588c2e1e7d6a7f92ea7a9fea5d07fedd5840c8b3365b664c3d701f1b325b65\": container with ID starting with eb588c2e1e7d6a7f92ea7a9fea5d07fedd5840c8b3365b664c3d701f1b325b65 not found: ID does not exist" containerID="eb588c2e1e7d6a7f92ea7a9fea5d07fedd5840c8b3365b664c3d701f1b325b65" Nov 02 23:57:19.591365 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:19.591305 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb588c2e1e7d6a7f92ea7a9fea5d07fedd5840c8b3365b664c3d701f1b325b65"} err="failed to get container status \"eb588c2e1e7d6a7f92ea7a9fea5d07fedd5840c8b3365b664c3d701f1b325b65\": rpc error: code = NotFound desc = could not find container \"eb588c2e1e7d6a7f92ea7a9fea5d07fedd5840c8b3365b664c3d701f1b325b65\": container with ID starting with eb588c2e1e7d6a7f92ea7a9fea5d07fedd5840c8b3365b664c3d701f1b325b65 not found: ID does not exist" Nov 02 23:57:19.591365 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:19.591327 5076 scope.go:122] "RemoveContainer" containerID="09fe8c125532ecb19785795e189dfc0e452a67429777d199981bd9c26eb2e8cb" Nov 02 23:57:19.591615 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:57:19.591599 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09fe8c125532ecb19785795e189dfc0e452a67429777d199981bd9c26eb2e8cb\": container with ID starting with 09fe8c125532ecb19785795e189dfc0e452a67429777d199981bd9c26eb2e8cb not found: ID does not exist" containerID="09fe8c125532ecb19785795e189dfc0e452a67429777d199981bd9c26eb2e8cb" Nov 02 23:57:19.591671 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:19.591622 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09fe8c125532ecb19785795e189dfc0e452a67429777d199981bd9c26eb2e8cb"} err="failed to get container status \"09fe8c125532ecb19785795e189dfc0e452a67429777d199981bd9c26eb2e8cb\": rpc error: code = NotFound desc = could not find container \"09fe8c125532ecb19785795e189dfc0e452a67429777d199981bd9c26eb2e8cb\": container with ID starting with 09fe8c125532ecb19785795e189dfc0e452a67429777d199981bd9c26eb2e8cb not found: ID does not exist" Nov 02 23:57:19.724854 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:19.724818 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2"] Nov 02 23:57:19.772398 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:19.772366 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-4b63d8-openshift-default-564c866b5-9clt2"] Nov 02 23:57:21.149069 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:21.149036 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fa13122-f046-4621-bf87-d124725da22f" path="/var/lib/kubelet/pods/9fa13122-f046-4621-bf87-d124725da22f/volumes" Nov 02 23:57:28.199999 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:28.199963 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6"] Nov 02 23:57:28.200511 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:28.200186 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" podUID="000f0ea7-6931-4670-9d4f-814c072405c9" containerName="istio-proxy" containerID="cri-o://a18ded61a21194f71ff35fc49cec0013c423eaf4133ebc338e45f095ac24f296" gracePeriod=30 Nov 02 23:57:32.014458 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:32.014424 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 02 23:57:32.073226 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:32.073199 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 02 23:57:33.479770 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.479744 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" Nov 02 23:57:33.625030 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.624996 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/000f0ea7-6931-4670-9d4f-814c072405c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/000f0ea7-6931-4670-9d4f-814c072405c9-credential-socket\") pod \"000f0ea7-6931-4670-9d4f-814c072405c9\" (UID: \"000f0ea7-6931-4670-9d4f-814c072405c9\") " Nov 02 23:57:33.625217 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.625074 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/000f0ea7-6931-4670-9d4f-814c072405c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/000f0ea7-6931-4670-9d4f-814c072405c9-istio-podinfo\") pod \"000f0ea7-6931-4670-9d4f-814c072405c9\" (UID: \"000f0ea7-6931-4670-9d4f-814c072405c9\") " Nov 02 23:57:33.625217 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.625133 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/000f0ea7-6931-4670-9d4f-814c072405c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/000f0ea7-6931-4670-9d4f-814c072405c9-istio-data\") pod \"000f0ea7-6931-4670-9d4f-814c072405c9\" (UID: \"000f0ea7-6931-4670-9d4f-814c072405c9\") " Nov 02 23:57:33.625217 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.625159 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/000f0ea7-6931-4670-9d4f-814c072405c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/000f0ea7-6931-4670-9d4f-814c072405c9-workload-socket\") pod \"000f0ea7-6931-4670-9d4f-814c072405c9\" (UID: \"000f0ea7-6931-4670-9d4f-814c072405c9\") " Nov 02 23:57:33.625404 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.625234 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/000f0ea7-6931-4670-9d4f-814c072405c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/000f0ea7-6931-4670-9d4f-814c072405c9-istio-envoy\") pod \"000f0ea7-6931-4670-9d4f-814c072405c9\" (UID: \"000f0ea7-6931-4670-9d4f-814c072405c9\") " Nov 02 23:57:33.625404 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.625280 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/000f0ea7-6931-4670-9d4f-814c072405c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/000f0ea7-6931-4670-9d4f-814c072405c9-istiod-ca-cert\") pod \"000f0ea7-6931-4670-9d4f-814c072405c9\" (UID: \"000f0ea7-6931-4670-9d4f-814c072405c9\") " Nov 02 23:57:33.625404 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.625303 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/000f0ea7-6931-4670-9d4f-814c072405c9-credential-socket" pod "000f0ea7-6931-4670-9d4f-814c072405c9" (UID: "000f0ea7-6931-4670-9d4f-814c072405c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:57:33.625404 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.625325 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/000f0ea7-6931-4670-9d4f-814c072405c9-istio-token\" (UniqueName: \"kubernetes.io/projected/000f0ea7-6931-4670-9d4f-814c072405c9-istio-token\") pod \"000f0ea7-6931-4670-9d4f-814c072405c9\" (UID: \"000f0ea7-6931-4670-9d4f-814c072405c9\") " Nov 02 23:57:33.625404 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.625403 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/000f0ea7-6931-4670-9d4f-814c072405c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/000f0ea7-6931-4670-9d4f-814c072405c9-workload-certs\") pod \"000f0ea7-6931-4670-9d4f-814c072405c9\" (UID: \"000f0ea7-6931-4670-9d4f-814c072405c9\") " Nov 02 23:57:33.625681 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.625421 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/000f0ea7-6931-4670-9d4f-814c072405c9-istio-data" pod "000f0ea7-6931-4670-9d4f-814c072405c9" (UID: "000f0ea7-6931-4670-9d4f-814c072405c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:57:33.625681 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.625493 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/000f0ea7-6931-4670-9d4f-814c072405c9-workload-socket" pod "000f0ea7-6931-4670-9d4f-814c072405c9" (UID: "000f0ea7-6931-4670-9d4f-814c072405c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:57:33.625681 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.625525 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/000f0ea7-6931-4670-9d4f-814c072405c9-kube-api-access-m98cd\" (UniqueName: \"kubernetes.io/projected/000f0ea7-6931-4670-9d4f-814c072405c9-kube-api-access-m98cd\") pod \"000f0ea7-6931-4670-9d4f-814c072405c9\" (UID: \"000f0ea7-6931-4670-9d4f-814c072405c9\") " Nov 02 23:57:33.625681 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.625661 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/000f0ea7-6931-4670-9d4f-814c072405c9-istiod-ca-cert" pod "000f0ea7-6931-4670-9d4f-814c072405c9" (UID: "000f0ea7-6931-4670-9d4f-814c072405c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:57:33.625902 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.625738 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/000f0ea7-6931-4670-9d4f-814c072405c9-workload-certs" pod "000f0ea7-6931-4670-9d4f-814c072405c9" (UID: "000f0ea7-6931-4670-9d4f-814c072405c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:57:33.626141 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.626045 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/000f0ea7-6931-4670-9d4f-814c072405c9-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:33.626141 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.626072 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/000f0ea7-6931-4670-9d4f-814c072405c9-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:33.626141 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.626091 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/000f0ea7-6931-4670-9d4f-814c072405c9-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:33.626141 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.626105 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/000f0ea7-6931-4670-9d4f-814c072405c9-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:33.626141 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.626119 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/000f0ea7-6931-4670-9d4f-814c072405c9-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:33.627606 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.627582 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/000f0ea7-6931-4670-9d4f-814c072405c9-istio-envoy" pod "000f0ea7-6931-4670-9d4f-814c072405c9" (UID: "000f0ea7-6931-4670-9d4f-814c072405c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:57:33.627697 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.627649 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/000f0ea7-6931-4670-9d4f-814c072405c9-istio-token" pod "000f0ea7-6931-4670-9d4f-814c072405c9" (UID: "000f0ea7-6931-4670-9d4f-814c072405c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:57:33.627763 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.627744 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/000f0ea7-6931-4670-9d4f-814c072405c9-istio-podinfo" pod "000f0ea7-6931-4670-9d4f-814c072405c9" (UID: "000f0ea7-6931-4670-9d4f-814c072405c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:57:33.627885 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.627870 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/000f0ea7-6931-4670-9d4f-814c072405c9-kube-api-access-m98cd" pod "000f0ea7-6931-4670-9d4f-814c072405c9" (UID: "000f0ea7-6931-4670-9d4f-814c072405c9"). InnerVolumeSpecName "kube-api-access-m98cd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:57:33.667436 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.667398 5076 generic.go:356] "Generic (PLEG): container finished" podID="000f0ea7-6931-4670-9d4f-814c072405c9" containerID="a18ded61a21194f71ff35fc49cec0013c423eaf4133ebc338e45f095ac24f296" exitCode=0 Nov 02 23:57:33.667635 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.667466 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" event={"ID":"000f0ea7-6931-4670-9d4f-814c072405c9","Type":"ContainerDied","Data":"a18ded61a21194f71ff35fc49cec0013c423eaf4133ebc338e45f095ac24f296"} Nov 02 23:57:33.667635 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.667537 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" event={"ID":"000f0ea7-6931-4670-9d4f-814c072405c9","Type":"ContainerDied","Data":"02a24b2390a7215e8767f2fecd5839e4a85bce93d10d94303a84d31a75f93b98"} Nov 02 23:57:33.667635 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.667467 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6" Nov 02 23:57:33.667635 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.667551 5076 scope.go:122] "RemoveContainer" containerID="a18ded61a21194f71ff35fc49cec0013c423eaf4133ebc338e45f095ac24f296" Nov 02 23:57:33.684733 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.684709 5076 scope.go:122] "RemoveContainer" containerID="7778c67442e41bf95c06a3aa3e62061243a047374935a420da52367cb9445e6c" Nov 02 23:57:33.695598 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.695580 5076 scope.go:122] "RemoveContainer" containerID="a18ded61a21194f71ff35fc49cec0013c423eaf4133ebc338e45f095ac24f296" Nov 02 23:57:33.695860 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:57:33.695830 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a18ded61a21194f71ff35fc49cec0013c423eaf4133ebc338e45f095ac24f296\": container with ID starting with a18ded61a21194f71ff35fc49cec0013c423eaf4133ebc338e45f095ac24f296 not found: ID does not exist" containerID="a18ded61a21194f71ff35fc49cec0013c423eaf4133ebc338e45f095ac24f296" Nov 02 23:57:33.695960 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.695861 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a18ded61a21194f71ff35fc49cec0013c423eaf4133ebc338e45f095ac24f296"} err="failed to get container status \"a18ded61a21194f71ff35fc49cec0013c423eaf4133ebc338e45f095ac24f296\": rpc error: code = NotFound desc = could not find container \"a18ded61a21194f71ff35fc49cec0013c423eaf4133ebc338e45f095ac24f296\": container with ID starting with a18ded61a21194f71ff35fc49cec0013c423eaf4133ebc338e45f095ac24f296 not found: ID does not exist" Nov 02 23:57:33.695960 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.695884 5076 scope.go:122] "RemoveContainer" containerID="7778c67442e41bf95c06a3aa3e62061243a047374935a420da52367cb9445e6c" Nov 02 23:57:33.696141 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:57:33.696116 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7778c67442e41bf95c06a3aa3e62061243a047374935a420da52367cb9445e6c\": container with ID starting with 7778c67442e41bf95c06a3aa3e62061243a047374935a420da52367cb9445e6c not found: ID does not exist" containerID="7778c67442e41bf95c06a3aa3e62061243a047374935a420da52367cb9445e6c" Nov 02 23:57:33.696213 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.696146 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7778c67442e41bf95c06a3aa3e62061243a047374935a420da52367cb9445e6c"} err="failed to get container status \"7778c67442e41bf95c06a3aa3e62061243a047374935a420da52367cb9445e6c\": rpc error: code = NotFound desc = could not find container \"7778c67442e41bf95c06a3aa3e62061243a047374935a420da52367cb9445e6c\": container with ID starting with 7778c67442e41bf95c06a3aa3e62061243a047374935a420da52367cb9445e6c not found: ID does not exist" Nov 02 23:57:33.727478 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.727422 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/000f0ea7-6931-4670-9d4f-814c072405c9-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:33.727478 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.727447 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m98cd\" (UniqueName: \"kubernetes.io/projected/000f0ea7-6931-4670-9d4f-814c072405c9-kube-api-access-m98cd\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:33.727478 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.727464 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/000f0ea7-6931-4670-9d4f-814c072405c9-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:33.727641 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.727504 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/000f0ea7-6931-4670-9d4f-814c072405c9-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:33.829017 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.828979 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6"] Nov 02 23:57:33.834807 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:33.834777 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-jzkx6"] Nov 02 23:57:35.148506 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:35.148466 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="000f0ea7-6931-4670-9d4f-814c072405c9" path="/var/lib/kubelet/pods/000f0ea7-6931-4670-9d4f-814c072405c9/volumes" Nov 02 23:57:39.883561 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:57:39.883522 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:57:45.580034 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.579950 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp"] Nov 02 23:57:45.580526 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.580509 5076 cpu_manager.go:414] "containerMap: removing container" podUID="000f0ea7-6931-4670-9d4f-814c072405c9" containerName="istio-proxy" Nov 02 23:57:45.580573 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.580527 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="000f0ea7-6931-4670-9d4f-814c072405c9" containerName="istio-proxy" Nov 02 23:57:45.580573 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.580535 5076 cpu_manager.go:414] "containerMap: removing container" podUID="c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa" containerName="maas-api" Nov 02 23:57:45.580573 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.580541 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa" containerName="maas-api" Nov 02 23:57:45.580573 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.580561 5076 cpu_manager.go:414] "containerMap: removing container" podUID="d2656fde-164f-40ec-845f-ad9c08091597" containerName="istio-proxy" Nov 02 23:57:45.580573 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.580566 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2656fde-164f-40ec-845f-ad9c08091597" containerName="istio-proxy" Nov 02 23:57:45.580735 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.580579 5076 cpu_manager.go:414] "containerMap: removing container" podUID="519907c3-2bc2-490b-a888-ee3b91ba5f8a" containerName="maas-api" Nov 02 23:57:45.580735 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.580583 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="519907c3-2bc2-490b-a888-ee3b91ba5f8a" containerName="maas-api" Nov 02 23:57:45.580735 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.580589 5076 cpu_manager.go:414] "containerMap: removing container" podUID="000f0ea7-6931-4670-9d4f-814c072405c9" containerName="istio-proxy" Nov 02 23:57:45.580735 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.580594 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="000f0ea7-6931-4670-9d4f-814c072405c9" containerName="istio-proxy" Nov 02 23:57:45.580735 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.580601 5076 cpu_manager.go:414] "containerMap: removing container" podUID="73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b" containerName="maas-api" Nov 02 23:57:45.580735 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.580606 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b" containerName="maas-api" Nov 02 23:57:45.580735 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.580618 5076 cpu_manager.go:414] "containerMap: removing container" podUID="9fa13122-f046-4621-bf87-d124725da22f" containerName="istio-proxy" Nov 02 23:57:45.580735 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.580624 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fa13122-f046-4621-bf87-d124725da22f" containerName="istio-proxy" Nov 02 23:57:45.580735 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.580632 5076 cpu_manager.go:414] "containerMap: removing container" podUID="7947bd91-9f2d-44dc-9e12-d3c8f3744ab0" containerName="istio-proxy" Nov 02 23:57:45.580735 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.580636 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7947bd91-9f2d-44dc-9e12-d3c8f3744ab0" containerName="istio-proxy" Nov 02 23:57:45.580735 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.580649 5076 cpu_manager.go:414] "containerMap: removing container" podUID="22a0ed3d-b728-4e61-81a2-1c118261632c" containerName="maas-api" Nov 02 23:57:45.580735 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.580653 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22a0ed3d-b728-4e61-81a2-1c118261632c" containerName="maas-api" Nov 02 23:57:45.580735 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.580660 5076 cpu_manager.go:414] "containerMap: removing container" podUID="9fa13122-f046-4621-bf87-d124725da22f" containerName="istio-proxy" Nov 02 23:57:45.580735 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.580665 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fa13122-f046-4621-bf87-d124725da22f" containerName="istio-proxy" Nov 02 23:57:45.580735 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.580675 5076 cpu_manager.go:414] "containerMap: removing container" podUID="7947bd91-9f2d-44dc-9e12-d3c8f3744ab0" containerName="istio-proxy" Nov 02 23:57:45.580735 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.580681 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7947bd91-9f2d-44dc-9e12-d3c8f3744ab0" containerName="istio-proxy" Nov 02 23:57:45.581253 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.580757 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="7947bd91-9f2d-44dc-9e12-d3c8f3744ab0" containerName="istio-proxy" Nov 02 23:57:45.581253 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.580767 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="000f0ea7-6931-4670-9d4f-814c072405c9" containerName="istio-proxy" Nov 02 23:57:45.581253 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.580774 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5e0e172-c639-4b9a-b2ad-fb3cb8f837aa" containerName="maas-api" Nov 02 23:57:45.581253 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.580781 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2656fde-164f-40ec-845f-ad9c08091597" containerName="istio-proxy" Nov 02 23:57:45.581253 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.580789 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="519907c3-2bc2-490b-a888-ee3b91ba5f8a" containerName="maas-api" Nov 02 23:57:45.581253 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.580795 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="73564eaf-3199-4a0b-8fc2-cbc7ee72eb9b" containerName="maas-api" Nov 02 23:57:45.581253 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.580802 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fa13122-f046-4621-bf87-d124725da22f" containerName="istio-proxy" Nov 02 23:57:45.581253 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.580808 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2656fde-164f-40ec-845f-ad9c08091597" containerName="istio-proxy" Nov 02 23:57:45.581253 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.580814 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="22a0ed3d-b728-4e61-81a2-1c118261632c" containerName="maas-api" Nov 02 23:57:45.581253 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.580820 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fa13122-f046-4621-bf87-d124725da22f" containerName="istio-proxy" Nov 02 23:57:45.583151 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.583132 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" Nov 02 23:57:45.708616 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.708573 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp"] Nov 02 23:57:45.713862 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.713834 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d4e172-af11-4e35-9363-c4959295db10-credential-socket\") pod \"e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp\" (UID: \"22d4e172-af11-4e35-9363-c4959295db10\") " pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" Nov 02 23:57:45.714180 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.714157 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22d4e172-af11-4e35-9363-c4959295db10-workload-certs\") pod \"e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp\" (UID: \"22d4e172-af11-4e35-9363-c4959295db10\") " pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" Nov 02 23:57:45.714293 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.714208 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22d4e172-af11-4e35-9363-c4959295db10-istio-envoy\") pod \"e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp\" (UID: \"22d4e172-af11-4e35-9363-c4959295db10\") " pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" Nov 02 23:57:45.714293 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.714262 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22d4e172-af11-4e35-9363-c4959295db10-istio-token\") pod \"e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp\" (UID: \"22d4e172-af11-4e35-9363-c4959295db10\") " pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" Nov 02 23:57:45.714411 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.714295 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsx76\" (UniqueName: \"kubernetes.io/projected/22d4e172-af11-4e35-9363-c4959295db10-kube-api-access-tsx76\") pod \"e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp\" (UID: \"22d4e172-af11-4e35-9363-c4959295db10\") " pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" Nov 02 23:57:45.714469 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.714410 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22d4e172-af11-4e35-9363-c4959295db10-istiod-ca-cert\") pod \"e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp\" (UID: \"22d4e172-af11-4e35-9363-c4959295db10\") " pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" Nov 02 23:57:45.714537 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.714520 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22d4e172-af11-4e35-9363-c4959295db10-istio-podinfo\") pod \"e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp\" (UID: \"22d4e172-af11-4e35-9363-c4959295db10\") " pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" Nov 02 23:57:45.714665 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.714648 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22d4e172-af11-4e35-9363-c4959295db10-istio-data\") pod \"e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp\" (UID: \"22d4e172-af11-4e35-9363-c4959295db10\") " pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" Nov 02 23:57:45.714749 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.714705 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d4e172-af11-4e35-9363-c4959295db10-workload-socket\") pod \"e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp\" (UID: \"22d4e172-af11-4e35-9363-c4959295db10\") " pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" Nov 02 23:57:45.753567 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.753534 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-ad18ae-openshift-default-dockercfg-9b966\"" Nov 02 23:57:45.815439 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.815406 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22d4e172-af11-4e35-9363-c4959295db10-istio-data\") pod \"e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp\" (UID: \"22d4e172-af11-4e35-9363-c4959295db10\") " pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" Nov 02 23:57:45.815636 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.815457 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d4e172-af11-4e35-9363-c4959295db10-workload-socket\") pod \"e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp\" (UID: \"22d4e172-af11-4e35-9363-c4959295db10\") " pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" Nov 02 23:57:45.815636 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.815491 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d4e172-af11-4e35-9363-c4959295db10-credential-socket\") pod \"e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp\" (UID: \"22d4e172-af11-4e35-9363-c4959295db10\") " pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" Nov 02 23:57:45.815636 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.815526 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22d4e172-af11-4e35-9363-c4959295db10-workload-certs\") pod \"e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp\" (UID: \"22d4e172-af11-4e35-9363-c4959295db10\") " pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" Nov 02 23:57:45.815636 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.815560 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22d4e172-af11-4e35-9363-c4959295db10-istio-envoy\") pod \"e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp\" (UID: \"22d4e172-af11-4e35-9363-c4959295db10\") " pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" Nov 02 23:57:45.815636 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.815608 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22d4e172-af11-4e35-9363-c4959295db10-istio-token\") pod \"e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp\" (UID: \"22d4e172-af11-4e35-9363-c4959295db10\") " pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" Nov 02 23:57:45.815636 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.815634 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsx76\" (UniqueName: \"kubernetes.io/projected/22d4e172-af11-4e35-9363-c4959295db10-kube-api-access-tsx76\") pod \"e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp\" (UID: \"22d4e172-af11-4e35-9363-c4959295db10\") " pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" Nov 02 23:57:45.816107 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.815810 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22d4e172-af11-4e35-9363-c4959295db10-istio-data\") pod \"e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp\" (UID: \"22d4e172-af11-4e35-9363-c4959295db10\") " pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" Nov 02 23:57:45.816107 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.815857 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d4e172-af11-4e35-9363-c4959295db10-workload-socket\") pod \"e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp\" (UID: \"22d4e172-af11-4e35-9363-c4959295db10\") " pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" Nov 02 23:57:45.816107 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.815912 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22d4e172-af11-4e35-9363-c4959295db10-istiod-ca-cert\") pod \"e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp\" (UID: \"22d4e172-af11-4e35-9363-c4959295db10\") " pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" Nov 02 23:57:45.816107 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.815944 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22d4e172-af11-4e35-9363-c4959295db10-workload-certs\") pod \"e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp\" (UID: \"22d4e172-af11-4e35-9363-c4959295db10\") " pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" Nov 02 23:57:45.816107 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.815978 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d4e172-af11-4e35-9363-c4959295db10-credential-socket\") pod \"e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp\" (UID: \"22d4e172-af11-4e35-9363-c4959295db10\") " pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" Nov 02 23:57:45.816107 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.816061 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22d4e172-af11-4e35-9363-c4959295db10-istio-podinfo\") pod \"e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp\" (UID: \"22d4e172-af11-4e35-9363-c4959295db10\") " pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" Nov 02 23:57:45.816486 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.816467 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22d4e172-af11-4e35-9363-c4959295db10-istiod-ca-cert\") pod \"e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp\" (UID: \"22d4e172-af11-4e35-9363-c4959295db10\") " pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" Nov 02 23:57:45.818076 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.818051 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22d4e172-af11-4e35-9363-c4959295db10-istio-envoy\") pod \"e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp\" (UID: \"22d4e172-af11-4e35-9363-c4959295db10\") " pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" Nov 02 23:57:45.818353 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.818335 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22d4e172-af11-4e35-9363-c4959295db10-istio-podinfo\") pod \"e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp\" (UID: \"22d4e172-af11-4e35-9363-c4959295db10\") " pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" Nov 02 23:57:45.870321 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.870247 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22d4e172-af11-4e35-9363-c4959295db10-istio-token\") pod \"e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp\" (UID: \"22d4e172-af11-4e35-9363-c4959295db10\") " pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" Nov 02 23:57:45.870446 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.870366 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsx76\" (UniqueName: \"kubernetes.io/projected/22d4e172-af11-4e35-9363-c4959295db10-kube-api-access-tsx76\") pod \"e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp\" (UID: \"22d4e172-af11-4e35-9363-c4959295db10\") " pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" Nov 02 23:57:45.895752 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:45.895715 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" Nov 02 23:57:46.335215 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:46.335187 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp"] Nov 02 23:57:46.336410 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:57:46.336381 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22d4e172_af11_4e35_9363_c4959295db10.slice/crio-0d8aeffde671ef9d4538322fc99c3dd0a5cfc2d48386dbe0b8702452840af46a WatchSource:0}: Error finding container 0d8aeffde671ef9d4538322fc99c3dd0a5cfc2d48386dbe0b8702452840af46a: Status 404 returned error can't find the container with id 0d8aeffde671ef9d4538322fc99c3dd0a5cfc2d48386dbe0b8702452840af46a Nov 02 23:57:46.338913 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:46.338857 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:57:46.338996 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:46.338953 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:57:46.338996 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:46.338986 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:57:46.765798 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:46.765763 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" event={"ID":"22d4e172-af11-4e35-9363-c4959295db10","Type":"ContainerStarted","Data":"a10016e2a412e13601247f08a90dbc56969d729dbac9493a49d0f680607503e5"} Nov 02 23:57:46.766296 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:46.765828 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" event={"ID":"22d4e172-af11-4e35-9363-c4959295db10","Type":"ContainerStarted","Data":"0d8aeffde671ef9d4538322fc99c3dd0a5cfc2d48386dbe0b8702452840af46a"} Nov 02 23:57:46.879422 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:46.879370 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" podStartSLOduration=1.879353847 podStartE2EDuration="1.879353847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:57:46.876669475 +0000 UTC m=+3915.845201259" watchObservedRunningTime="2026-11-02 23:57:46.879353847 +0000 UTC m=+3915.847885631" Nov 02 23:57:46.896002 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:46.895974 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" Nov 02 23:57:47.148254 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:47.148178 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm"] Nov 02 23:57:47.148415 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:47.148369 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm" podUID="efd74f02-817e-4143-a98d-6d82d8fc3e2b" containerName="maas-api" containerID="cri-o://1f54a180dd7da9d2d29c8b0b69c7566702577d5436b03ade0b77ce649978a751" gracePeriod=30 Nov 02 23:57:47.455232 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:47.455203 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm" Nov 02 23:57:47.539321 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:47.539281 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efd74f02-817e-4143-a98d-6d82d8fc3e2b-kube-api-access-kfl88\" (UniqueName: \"kubernetes.io/projected/efd74f02-817e-4143-a98d-6d82d8fc3e2b-kube-api-access-kfl88\") pod \"efd74f02-817e-4143-a98d-6d82d8fc3e2b\" (UID: \"efd74f02-817e-4143-a98d-6d82d8fc3e2b\") " Nov 02 23:57:47.540130 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:47.540091 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efd74f02-817e-4143-a98d-6d82d8fc3e2b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efd74f02-817e-4143-a98d-6d82d8fc3e2b-maas-api-tls\") pod \"efd74f02-817e-4143-a98d-6d82d8fc3e2b\" (UID: \"efd74f02-817e-4143-a98d-6d82d8fc3e2b\") " Nov 02 23:57:47.540727 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:47.540702 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efd74f02-817e-4143-a98d-6d82d8fc3e2b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efd74f02-817e-4143-a98d-6d82d8fc3e2b-openshift-service-ca\") pod \"efd74f02-817e-4143-a98d-6d82d8fc3e2b\" (UID: \"efd74f02-817e-4143-a98d-6d82d8fc3e2b\") " Nov 02 23:57:47.553288 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:47.553259 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efd74f02-817e-4143-a98d-6d82d8fc3e2b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/efd74f02-817e-4143-a98d-6d82d8fc3e2b-metrics-certs\") pod \"efd74f02-817e-4143-a98d-6d82d8fc3e2b\" (UID: \"efd74f02-817e-4143-a98d-6d82d8fc3e2b\") " Nov 02 23:57:47.557518 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:47.553023 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efd74f02-817e-4143-a98d-6d82d8fc3e2b-openshift-service-ca" pod "efd74f02-817e-4143-a98d-6d82d8fc3e2b" (UID: "efd74f02-817e-4143-a98d-6d82d8fc3e2b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:57:47.557652 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:47.557625 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efd74f02-817e-4143-a98d-6d82d8fc3e2b-maas-api-tls" pod "efd74f02-817e-4143-a98d-6d82d8fc3e2b" (UID: "efd74f02-817e-4143-a98d-6d82d8fc3e2b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:57:47.558469 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:47.558410 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efd74f02-817e-4143-a98d-6d82d8fc3e2b-kube-api-access-kfl88" pod "efd74f02-817e-4143-a98d-6d82d8fc3e2b" (UID: "efd74f02-817e-4143-a98d-6d82d8fc3e2b"). InnerVolumeSpecName "kube-api-access-kfl88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:57:47.559647 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:47.559212 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efd74f02-817e-4143-a98d-6d82d8fc3e2b-metrics-certs" pod "efd74f02-817e-4143-a98d-6d82d8fc3e2b" (UID: "efd74f02-817e-4143-a98d-6d82d8fc3e2b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:57:47.655284 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:47.655242 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efd74f02-817e-4143-a98d-6d82d8fc3e2b-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:47.655284 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:47.655276 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfl88\" (UniqueName: \"kubernetes.io/projected/efd74f02-817e-4143-a98d-6d82d8fc3e2b-kube-api-access-kfl88\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:47.655284 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:47.655290 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efd74f02-817e-4143-a98d-6d82d8fc3e2b-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:47.655522 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:47.655301 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efd74f02-817e-4143-a98d-6d82d8fc3e2b-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:47.772535 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:47.772504 5076 generic.go:356] "Generic (PLEG): container finished" podID="efd74f02-817e-4143-a98d-6d82d8fc3e2b" containerID="1f54a180dd7da9d2d29c8b0b69c7566702577d5436b03ade0b77ce649978a751" exitCode=0 Nov 02 23:57:47.772990 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:47.772566 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm" Nov 02 23:57:47.772990 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:47.772593 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm" event={"ID":"efd74f02-817e-4143-a98d-6d82d8fc3e2b","Type":"ContainerDied","Data":"1f54a180dd7da9d2d29c8b0b69c7566702577d5436b03ade0b77ce649978a751"} Nov 02 23:57:47.772990 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:47.772648 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm" event={"ID":"efd74f02-817e-4143-a98d-6d82d8fc3e2b","Type":"ContainerDied","Data":"c543fe354231e4c2efca8272e894fac434c965d93405fabaf6e1921c0b6defbb"} Nov 02 23:57:47.772990 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:47.772670 5076 scope.go:122] "RemoveContainer" containerID="1f54a180dd7da9d2d29c8b0b69c7566702577d5436b03ade0b77ce649978a751" Nov 02 23:57:47.782973 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:47.782958 5076 scope.go:122] "RemoveContainer" containerID="1f54a180dd7da9d2d29c8b0b69c7566702577d5436b03ade0b77ce649978a751" Nov 02 23:57:47.783269 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:57:47.783249 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f54a180dd7da9d2d29c8b0b69c7566702577d5436b03ade0b77ce649978a751\": container with ID starting with 1f54a180dd7da9d2d29c8b0b69c7566702577d5436b03ade0b77ce649978a751 not found: ID does not exist" containerID="1f54a180dd7da9d2d29c8b0b69c7566702577d5436b03ade0b77ce649978a751" Nov 02 23:57:47.783339 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:47.783279 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f54a180dd7da9d2d29c8b0b69c7566702577d5436b03ade0b77ce649978a751"} err="failed to get container status \"1f54a180dd7da9d2d29c8b0b69c7566702577d5436b03ade0b77ce649978a751\": rpc error: code = NotFound desc = could not find container \"1f54a180dd7da9d2d29c8b0b69c7566702577d5436b03ade0b77ce649978a751\": container with ID starting with 1f54a180dd7da9d2d29c8b0b69c7566702577d5436b03ade0b77ce649978a751 not found: ID does not exist" Nov 02 23:57:47.902395 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:47.902362 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" Nov 02 23:57:47.930066 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:47.930035 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm"] Nov 02 23:57:47.977127 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:47.977100 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e65845-7fd77b9fc6-rl8rm"] Nov 02 23:57:48.780554 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:48.780514 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" Nov 02 23:57:48.781794 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:48.781769 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" Nov 02 23:57:49.158651 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:49.158558 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efd74f02-817e-4143-a98d-6d82d8fc3e2b" path="/var/lib/kubelet/pods/efd74f02-817e-4143-a98d-6d82d8fc3e2b/volumes" Nov 02 23:57:59.391378 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:59.391343 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb"] Nov 02 23:57:59.391919 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:59.391883 5076 cpu_manager.go:414] "containerMap: removing container" podUID="efd74f02-817e-4143-a98d-6d82d8fc3e2b" containerName="maas-api" Nov 02 23:57:59.391919 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:59.391912 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efd74f02-817e-4143-a98d-6d82d8fc3e2b" containerName="maas-api" Nov 02 23:57:59.392251 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:59.391937 5076 cpu_manager.go:414] "containerMap: removing container" podUID="d2656fde-164f-40ec-845f-ad9c08091597" containerName="istio-proxy" Nov 02 23:57:59.392251 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:59.391942 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2656fde-164f-40ec-845f-ad9c08091597" containerName="istio-proxy" Nov 02 23:57:59.392251 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:59.392033 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="efd74f02-817e-4143-a98d-6d82d8fc3e2b" containerName="maas-api" Nov 02 23:57:59.392251 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:59.392041 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="7947bd91-9f2d-44dc-9e12-d3c8f3744ab0" containerName="istio-proxy" Nov 02 23:57:59.392251 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:59.392049 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="000f0ea7-6931-4670-9d4f-814c072405c9" containerName="istio-proxy" Nov 02 23:57:59.396976 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:59.396955 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" Nov 02 23:57:59.431119 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:59.431086 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-istio-envoy\") pod \"e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb\" (UID: \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\") " pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" Nov 02 23:57:59.431299 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:59.431196 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-workload-certs\") pod \"e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb\" (UID: \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\") " pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" Nov 02 23:57:59.431299 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:59.431263 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-istio-data\") pod \"e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb\" (UID: \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\") " pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" Nov 02 23:57:59.431431 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:59.431307 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c9mn\" (UniqueName: \"kubernetes.io/projected/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-kube-api-access-6c9mn\") pod \"e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb\" (UID: \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\") " pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" Nov 02 23:57:59.431431 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:59.431394 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-istio-token\") pod \"e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb\" (UID: \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\") " pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" Nov 02 23:57:59.431431 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:59.431424 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-istiod-ca-cert\") pod \"e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb\" (UID: \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\") " pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" Nov 02 23:57:59.431658 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:59.431628 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-istio-podinfo\") pod \"e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb\" (UID: \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\") " pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" Nov 02 23:57:59.431704 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:59.431678 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-credential-socket\") pod \"e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb\" (UID: \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\") " pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" Nov 02 23:57:59.431806 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:59.431775 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-workload-socket\") pod \"e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb\" (UID: \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\") " pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" Nov 02 23:57:59.446698 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:59.446671 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e1be81-openshift-default-dockercfg-ws2q7\"" Nov 02 23:57:59.498187 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:59.498155 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb"] Nov 02 23:57:59.542534 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:59.542496 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-istio-envoy\") pod \"e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb\" (UID: \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\") " pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" Nov 02 23:57:59.543845 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:59.543815 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-workload-certs\") pod \"e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb\" (UID: \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\") " pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" Nov 02 23:57:59.545021 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:59.544686 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-istio-data\") pod \"e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb\" (UID: \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\") " pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" Nov 02 23:57:59.545219 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:59.545200 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c9mn\" (UniqueName: \"kubernetes.io/projected/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-kube-api-access-6c9mn\") pod \"e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb\" (UID: \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\") " pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" Nov 02 23:57:59.545719 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:59.545699 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-istio-token\") pod \"e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb\" (UID: \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\") " pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" Nov 02 23:57:59.546259 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:59.546241 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-istiod-ca-cert\") pod \"e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb\" (UID: \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\") " pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" Nov 02 23:57:59.547355 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:59.546196 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-istio-data\") pod \"e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb\" (UID: \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\") " pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" Nov 02 23:57:59.547355 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:59.544636 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-workload-certs\") pod \"e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb\" (UID: \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\") " pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" Nov 02 23:57:59.574914 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:59.558085 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-istio-podinfo\") pod \"e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb\" (UID: \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\") " pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" Nov 02 23:57:59.574914 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:59.558143 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-credential-socket\") pod \"e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb\" (UID: \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\") " pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" Nov 02 23:57:59.574914 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:59.558175 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-workload-socket\") pod \"e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb\" (UID: \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\") " pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" Nov 02 23:57:59.574914 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:59.558591 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-workload-socket\") pod \"e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb\" (UID: \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\") " pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" Nov 02 23:57:59.574914 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:59.559265 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-credential-socket\") pod \"e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb\" (UID: \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\") " pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" Nov 02 23:57:59.575764 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:59.575738 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-istiod-ca-cert\") pod \"e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb\" (UID: \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\") " pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" Nov 02 23:57:59.615554 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:59.614620 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-istio-podinfo\") pod \"e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb\" (UID: \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\") " pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" Nov 02 23:57:59.618670 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:59.617212 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-istio-envoy\") pod \"e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb\" (UID: \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\") " pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" Nov 02 23:57:59.618860 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:59.618607 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-istio-token\") pod \"e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb\" (UID: \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\") " pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" Nov 02 23:57:59.625340 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:59.625313 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c9mn\" (UniqueName: \"kubernetes.io/projected/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-kube-api-access-6c9mn\") pod \"e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb\" (UID: \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\") " pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" Nov 02 23:57:59.710653 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:57:59.710626 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" Nov 02 23:58:00.136767 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:00.136629 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb"] Nov 02 23:58:00.144957 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:00.144919 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:58:00.145117 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:00.145034 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:58:00.145117 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:00.145085 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:58:00.874418 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:00.874372 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" event={"ID":"df94f65b-8ef5-4e2c-a749-93a97b9c68e0","Type":"ContainerStarted","Data":"d99611eee0218a1928e4a81cd7dc524b3fd2340685a3fefc42c6c1cd09bfc6bd"} Nov 02 23:58:00.874932 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:00.874443 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" event={"ID":"df94f65b-8ef5-4e2c-a749-93a97b9c68e0","Type":"ContainerStarted","Data":"0843c2b2b2c9949259a6c559ab1b9e3e18fc8413afe3471c31b51510fde88804"} Nov 02 23:58:01.023815 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:01.023760 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" podStartSLOduration=2.02374375 podStartE2EDuration="2.02374375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:58:01.022964082 +0000 UTC m=+3929.991495868" watchObservedRunningTime="2026-11-02 23:58:01.02374375 +0000 UTC m=+3929.992275599" Nov 02 23:58:01.711079 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:01.711042 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" Nov 02 23:58:01.716868 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:01.716845 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" Nov 02 23:58:01.880156 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:01.880125 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" Nov 02 23:58:01.881112 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:01.881088 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" Nov 02 23:58:08.346488 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:08.346451 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p"] Nov 02 23:58:08.350483 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:08.350455 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" Nov 02 23:58:08.400789 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:08.400761 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-tbbgz\"" Nov 02 23:58:08.452358 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:08.452325 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p"] Nov 02 23:58:08.472404 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:08.472369 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b45216a5-f1ba-4744-bd22-c59437513e86-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vx88p\" (UID: \"b45216a5-f1ba-4744-bd22-c59437513e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" Nov 02 23:58:08.472557 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:08.472517 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45216a5-f1ba-4744-bd22-c59437513e86-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vx88p\" (UID: \"b45216a5-f1ba-4744-bd22-c59437513e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" Nov 02 23:58:08.472601 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:08.472560 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45216a5-f1ba-4744-bd22-c59437513e86-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vx88p\" (UID: \"b45216a5-f1ba-4744-bd22-c59437513e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" Nov 02 23:58:08.472698 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:08.472673 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b45216a5-f1ba-4744-bd22-c59437513e86-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vx88p\" (UID: \"b45216a5-f1ba-4744-bd22-c59437513e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" Nov 02 23:58:08.472750 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:08.472733 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b45216a5-f1ba-4744-bd22-c59437513e86-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vx88p\" (UID: \"b45216a5-f1ba-4744-bd22-c59437513e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" Nov 02 23:58:08.472794 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:08.472780 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j86gn\" (UniqueName: \"kubernetes.io/projected/b45216a5-f1ba-4744-bd22-c59437513e86-kube-api-access-j86gn\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vx88p\" (UID: \"b45216a5-f1ba-4744-bd22-c59437513e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" Nov 02 23:58:08.472851 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:08.472837 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b45216a5-f1ba-4744-bd22-c59437513e86-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vx88p\" (UID: \"b45216a5-f1ba-4744-bd22-c59437513e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" Nov 02 23:58:08.472978 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:08.472956 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b45216a5-f1ba-4744-bd22-c59437513e86-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vx88p\" (UID: \"b45216a5-f1ba-4744-bd22-c59437513e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" Nov 02 23:58:08.473025 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:08.473002 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b45216a5-f1ba-4744-bd22-c59437513e86-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vx88p\" (UID: \"b45216a5-f1ba-4744-bd22-c59437513e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" Nov 02 23:58:08.574186 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:08.574152 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b45216a5-f1ba-4744-bd22-c59437513e86-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vx88p\" (UID: \"b45216a5-f1ba-4744-bd22-c59437513e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" Nov 02 23:58:08.574355 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:08.574243 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45216a5-f1ba-4744-bd22-c59437513e86-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vx88p\" (UID: \"b45216a5-f1ba-4744-bd22-c59437513e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" Nov 02 23:58:08.574355 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:08.574265 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45216a5-f1ba-4744-bd22-c59437513e86-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vx88p\" (UID: \"b45216a5-f1ba-4744-bd22-c59437513e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" Nov 02 23:58:08.574355 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:08.574294 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b45216a5-f1ba-4744-bd22-c59437513e86-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vx88p\" (UID: \"b45216a5-f1ba-4744-bd22-c59437513e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" Nov 02 23:58:08.574355 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:08.574321 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b45216a5-f1ba-4744-bd22-c59437513e86-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vx88p\" (UID: \"b45216a5-f1ba-4744-bd22-c59437513e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" Nov 02 23:58:08.574590 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:08.574513 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j86gn\" (UniqueName: \"kubernetes.io/projected/b45216a5-f1ba-4744-bd22-c59437513e86-kube-api-access-j86gn\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vx88p\" (UID: \"b45216a5-f1ba-4744-bd22-c59437513e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" Nov 02 23:58:08.574590 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:08.574567 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b45216a5-f1ba-4744-bd22-c59437513e86-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vx88p\" (UID: \"b45216a5-f1ba-4744-bd22-c59437513e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" Nov 02 23:58:08.574703 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:08.574627 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b45216a5-f1ba-4744-bd22-c59437513e86-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vx88p\" (UID: \"b45216a5-f1ba-4744-bd22-c59437513e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" Nov 02 23:58:08.574703 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:08.574639 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b45216a5-f1ba-4744-bd22-c59437513e86-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vx88p\" (UID: \"b45216a5-f1ba-4744-bd22-c59437513e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" Nov 02 23:58:08.574703 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:08.574656 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45216a5-f1ba-4744-bd22-c59437513e86-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vx88p\" (UID: \"b45216a5-f1ba-4744-bd22-c59437513e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" Nov 02 23:58:08.574703 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:08.574662 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b45216a5-f1ba-4744-bd22-c59437513e86-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vx88p\" (UID: \"b45216a5-f1ba-4744-bd22-c59437513e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" Nov 02 23:58:08.575121 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:08.575096 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b45216a5-f1ba-4744-bd22-c59437513e86-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vx88p\" (UID: \"b45216a5-f1ba-4744-bd22-c59437513e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" Nov 02 23:58:08.575247 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:08.575127 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b45216a5-f1ba-4744-bd22-c59437513e86-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vx88p\" (UID: \"b45216a5-f1ba-4744-bd22-c59437513e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" Nov 02 23:58:08.575247 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:08.575178 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45216a5-f1ba-4744-bd22-c59437513e86-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vx88p\" (UID: \"b45216a5-f1ba-4744-bd22-c59437513e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" Nov 02 23:58:08.577187 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:08.577165 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b45216a5-f1ba-4744-bd22-c59437513e86-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vx88p\" (UID: \"b45216a5-f1ba-4744-bd22-c59437513e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" Nov 02 23:58:08.577330 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:08.577310 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b45216a5-f1ba-4744-bd22-c59437513e86-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vx88p\" (UID: \"b45216a5-f1ba-4744-bd22-c59437513e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" Nov 02 23:58:08.628939 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:08.628842 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b45216a5-f1ba-4744-bd22-c59437513e86-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vx88p\" (UID: \"b45216a5-f1ba-4744-bd22-c59437513e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" Nov 02 23:58:08.629073 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:08.629046 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j86gn\" (UniqueName: \"kubernetes.io/projected/b45216a5-f1ba-4744-bd22-c59437513e86-kube-api-access-j86gn\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vx88p\" (UID: \"b45216a5-f1ba-4744-bd22-c59437513e86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" Nov 02 23:58:08.665381 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:08.665350 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" Nov 02 23:58:09.150738 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:09.136878 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p"] Nov 02 23:58:09.152933 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:09.152457 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:58:09.152933 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:09.152534 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:58:09.152933 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:09.152565 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:58:09.884097 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:58:09.884065 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:58:09.953861 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:09.953829 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" event={"ID":"b45216a5-f1ba-4744-bd22-c59437513e86","Type":"ContainerStarted","Data":"de90b89e5e6cd36d6e364b767496267f914aeb7089b9aa8ce383c4453f1a5d07"} Nov 02 23:58:09.954048 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:09.953939 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" event={"ID":"b45216a5-f1ba-4744-bd22-c59437513e86","Type":"ContainerStarted","Data":"7d8b14eb9ecf7a25de77bc9439afb588c0e36230a0f1eace3d8ad482b46823bf"} Nov 02 23:58:10.665945 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:10.665904 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" Nov 02 23:58:10.667572 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:10.667537 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-vx88p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 02 23:58:10.667699 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:10.667600 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" podUID="b45216a5-f1ba-4744-bd22-c59437513e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 02 23:58:11.666265 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:11.666227 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-vx88p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 02 23:58:11.666676 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:11.666279 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" podUID="b45216a5-f1ba-4744-bd22-c59437513e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 02 23:58:12.666134 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:12.666099 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-vx88p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 02 23:58:12.666292 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:12.666156 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" podUID="b45216a5-f1ba-4744-bd22-c59437513e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 02 23:58:13.666260 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:13.666220 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-vx88p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 02 23:58:13.666475 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:13.666282 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" podUID="b45216a5-f1ba-4744-bd22-c59437513e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 02 23:58:14.666691 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:14.666656 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-vx88p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 02 23:58:14.667130 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:14.666710 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" podUID="b45216a5-f1ba-4744-bd22-c59437513e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 02 23:58:15.666413 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:15.666378 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-vx88p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 02 23:58:15.666570 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:15.666428 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" podUID="b45216a5-f1ba-4744-bd22-c59437513e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 02 23:58:16.666185 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:16.666143 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-vx88p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 02 23:58:16.666704 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:16.666197 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" podUID="b45216a5-f1ba-4744-bd22-c59437513e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 02 23:58:17.666241 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:17.666204 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-vx88p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 02 23:58:17.666775 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:17.666260 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" podUID="b45216a5-f1ba-4744-bd22-c59437513e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 02 23:58:19.666939 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:19.666883 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-vx88p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:58:19.667460 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:19.666964 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" podUID="b45216a5-f1ba-4744-bd22-c59437513e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:58:20.708048 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:20.708012 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-vx88p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:58:20.708529 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:20.708073 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" podUID="b45216a5-f1ba-4744-bd22-c59437513e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:58:21.074798 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:21.074708 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" Nov 02 23:58:21.075100 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:21.075078 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" Nov 02 23:58:21.075733 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:21.075712 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" Nov 02 23:58:21.235818 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:21.235767 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" podStartSLOduration=13.23575249 podStartE2EDuration="13.23575249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:58:10.163040749 +0000 UTC m=+3939.131572532" watchObservedRunningTime="2026-11-02 23:58:21.23575249 +0000 UTC m=+3950.204284482" Nov 02 23:58:39.884633 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:58:39.884599 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:58:58.665917 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:58.665872 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg"] Nov 02 23:58:58.670036 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:58.670016 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg" Nov 02 23:58:58.720259 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:58.720232 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-ad18ae-serving-cert\"" Nov 02 23:58:58.770386 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:58.770355 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg"] Nov 02 23:58:58.787071 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:58.787043 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3533e1bf-11a4-4c94-a310-6edeaa98a834-metrics-certs\") pod \"maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg\" (UID: \"3533e1bf-11a4-4c94-a310-6edeaa98a834\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg" Nov 02 23:58:58.787221 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:58.787083 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3533e1bf-11a4-4c94-a310-6edeaa98a834-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg\" (UID: \"3533e1bf-11a4-4c94-a310-6edeaa98a834\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg" Nov 02 23:58:58.787269 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:58.787217 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3533e1bf-11a4-4c94-a310-6edeaa98a834-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg\" (UID: \"3533e1bf-11a4-4c94-a310-6edeaa98a834\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg" Nov 02 23:58:58.787269 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:58.787252 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwbq2\" (UniqueName: \"kubernetes.io/projected/3533e1bf-11a4-4c94-a310-6edeaa98a834-kube-api-access-fwbq2\") pod \"maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg\" (UID: \"3533e1bf-11a4-4c94-a310-6edeaa98a834\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg" Nov 02 23:58:58.887935 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:58.887878 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3533e1bf-11a4-4c94-a310-6edeaa98a834-metrics-certs\") pod \"maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg\" (UID: \"3533e1bf-11a4-4c94-a310-6edeaa98a834\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg" Nov 02 23:58:58.888106 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:58.887962 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3533e1bf-11a4-4c94-a310-6edeaa98a834-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg\" (UID: \"3533e1bf-11a4-4c94-a310-6edeaa98a834\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg" Nov 02 23:58:58.888106 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:58.888080 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3533e1bf-11a4-4c94-a310-6edeaa98a834-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg\" (UID: \"3533e1bf-11a4-4c94-a310-6edeaa98a834\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg" Nov 02 23:58:58.888106 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:58:58.888095 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad18ae-serving-cert: secret "maas-api-e2e-extmod-a-ad18ae-serving-cert" not found Nov 02 23:58:58.888219 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:58.888114 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwbq2\" (UniqueName: \"kubernetes.io/projected/3533e1bf-11a4-4c94-a310-6edeaa98a834-kube-api-access-fwbq2\") pod \"maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg\" (UID: \"3533e1bf-11a4-4c94-a310-6edeaa98a834\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg" Nov 02 23:58:58.888219 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:58:58.888166 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3533e1bf-11a4-4c94-a310-6edeaa98a834-maas-api-tls podName:3533e1bf-11a4-4c94-a310-6edeaa98a834 nodeName:}" failed. No retries permitted until 2026-11-02 23:58:59.388149505 +0000 UTC m=+3988.356681266 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3533e1bf-11a4-4c94-a310-6edeaa98a834-maas-api-tls") pod "maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg" (UID: "3533e1bf-11a4-4c94-a310-6edeaa98a834") : secret "maas-api-e2e-extmod-a-ad18ae-serving-cert" not found Nov 02 23:58:58.888712 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:58.888689 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3533e1bf-11a4-4c94-a310-6edeaa98a834-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg\" (UID: \"3533e1bf-11a4-4c94-a310-6edeaa98a834\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg" Nov 02 23:58:58.890367 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:58.890347 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3533e1bf-11a4-4c94-a310-6edeaa98a834-metrics-certs\") pod \"maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg\" (UID: \"3533e1bf-11a4-4c94-a310-6edeaa98a834\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg" Nov 02 23:58:58.943554 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:58.943517 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwbq2\" (UniqueName: \"kubernetes.io/projected/3533e1bf-11a4-4c94-a310-6edeaa98a834-kube-api-access-fwbq2\") pod \"maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg\" (UID: \"3533e1bf-11a4-4c94-a310-6edeaa98a834\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg" Nov 02 23:58:59.395042 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:59.394952 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3533e1bf-11a4-4c94-a310-6edeaa98a834-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg\" (UID: \"3533e1bf-11a4-4c94-a310-6edeaa98a834\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg" Nov 02 23:58:59.397419 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:59.397394 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3533e1bf-11a4-4c94-a310-6edeaa98a834-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg\" (UID: \"3533e1bf-11a4-4c94-a310-6edeaa98a834\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg" Nov 02 23:58:59.583680 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:59.583645 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg" Nov 02 23:58:59.950492 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:59.950462 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg"] Nov 02 23:58:59.951207 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:58:59.951176 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3533e1bf_11a4_4c94_a310_6edeaa98a834.slice/crio-8dc8d7deebae50390d4f1fa8880e889378201a0b96a6fb8a4eb16a09205ee77c WatchSource:0}: Error finding container 8dc8d7deebae50390d4f1fa8880e889378201a0b96a6fb8a4eb16a09205ee77c: Status 404 returned error can't find the container with id 8dc8d7deebae50390d4f1fa8880e889378201a0b96a6fb8a4eb16a09205ee77c Nov 02 23:58:59.952578 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:58:59.952560 5076 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 23:59:00.271140 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:00.271054 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg" event={"ID":"3533e1bf-11a4-4c94-a310-6edeaa98a834","Type":"ContainerStarted","Data":"8dc8d7deebae50390d4f1fa8880e889378201a0b96a6fb8a4eb16a09205ee77c"} Nov 02 23:59:01.279110 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:01.279013 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg" event={"ID":"3533e1bf-11a4-4c94-a310-6edeaa98a834","Type":"ContainerStarted","Data":"a675e8415013454b1ec975ce57e8e69599803b3f4b4e790ed2e4d5dae3741bfc"} Nov 02 23:59:01.279110 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:01.279099 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg" Nov 02 23:59:01.391829 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:01.391775 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg" podStartSLOduration=2.396237335 podStartE2EDuration="3.391760855s" totalImagesPullingTime="995.52352ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:58:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:59:01.391047136 +0000 UTC m=+3990.359579027" watchObservedRunningTime="2026-11-02 23:59:01.391760855 +0000 UTC m=+3990.360292638" Nov 02 23:59:07.289705 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:07.289678 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg" Nov 02 23:59:09.884603 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:59:09.884558 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:59:13.483071 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:13.482990 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd"] Nov 02 23:59:13.487108 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:13.487092 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd" Nov 02 23:59:13.540985 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:13.540956 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd"] Nov 02 23:59:13.581196 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:13.581168 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a09ac6d-1d69-4d01-8863-5e693b5533df-maas-api-tls\") pod \"maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd\" (UID: \"9a09ac6d-1d69-4d01-8863-5e693b5533df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd" Nov 02 23:59:13.581385 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:13.581270 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a09ac6d-1d69-4d01-8863-5e693b5533df-metrics-certs\") pod \"maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd\" (UID: \"9a09ac6d-1d69-4d01-8863-5e693b5533df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd" Nov 02 23:59:13.581385 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:13.581358 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a09ac6d-1d69-4d01-8863-5e693b5533df-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd\" (UID: \"9a09ac6d-1d69-4d01-8863-5e693b5533df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd" Nov 02 23:59:13.581525 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:13.581421 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhqb9\" (UniqueName: \"kubernetes.io/projected/9a09ac6d-1d69-4d01-8863-5e693b5533df-kube-api-access-jhqb9\") pod \"maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd\" (UID: \"9a09ac6d-1d69-4d01-8863-5e693b5533df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd" Nov 02 23:59:13.584434 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:13.584409 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e1be81-serving-cert\"" Nov 02 23:59:13.683151 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:13.683119 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a09ac6d-1d69-4d01-8863-5e693b5533df-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd\" (UID: \"9a09ac6d-1d69-4d01-8863-5e693b5533df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd" Nov 02 23:59:13.683313 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:13.683214 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhqb9\" (UniqueName: \"kubernetes.io/projected/9a09ac6d-1d69-4d01-8863-5e693b5533df-kube-api-access-jhqb9\") pod \"maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd\" (UID: \"9a09ac6d-1d69-4d01-8863-5e693b5533df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd" Nov 02 23:59:13.683313 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:13.683299 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a09ac6d-1d69-4d01-8863-5e693b5533df-maas-api-tls\") pod \"maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd\" (UID: \"9a09ac6d-1d69-4d01-8863-5e693b5533df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd" Nov 02 23:59:13.683397 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:13.683376 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a09ac6d-1d69-4d01-8863-5e693b5533df-metrics-certs\") pod \"maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd\" (UID: \"9a09ac6d-1d69-4d01-8863-5e693b5533df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd" Nov 02 23:59:13.683447 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:59:13.683430 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-e1be81-serving-cert: secret "maas-api-e2e-inf-a-e1be81-serving-cert" not found Nov 02 23:59:13.683518 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:59:13.683502 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a09ac6d-1d69-4d01-8863-5e693b5533df-maas-api-tls podName:9a09ac6d-1d69-4d01-8863-5e693b5533df nodeName:}" failed. No retries permitted until 2026-11-02 23:59:14.183483162 +0000 UTC m=+4003.152014930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9a09ac6d-1d69-4d01-8863-5e693b5533df-maas-api-tls") pod "maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd" (UID: "9a09ac6d-1d69-4d01-8863-5e693b5533df") : secret "maas-api-e2e-inf-a-e1be81-serving-cert" not found Nov 02 23:59:13.683835 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:13.683817 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a09ac6d-1d69-4d01-8863-5e693b5533df-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd\" (UID: \"9a09ac6d-1d69-4d01-8863-5e693b5533df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd" Nov 02 23:59:13.685863 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:13.685837 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a09ac6d-1d69-4d01-8863-5e693b5533df-metrics-certs\") pod \"maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd\" (UID: \"9a09ac6d-1d69-4d01-8863-5e693b5533df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd" Nov 02 23:59:13.737850 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:13.737784 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhqb9\" (UniqueName: \"kubernetes.io/projected/9a09ac6d-1d69-4d01-8863-5e693b5533df-kube-api-access-jhqb9\") pod \"maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd\" (UID: \"9a09ac6d-1d69-4d01-8863-5e693b5533df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd" Nov 02 23:59:14.195765 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:14.195730 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a09ac6d-1d69-4d01-8863-5e693b5533df-maas-api-tls\") pod \"maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd\" (UID: \"9a09ac6d-1d69-4d01-8863-5e693b5533df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd" Nov 02 23:59:14.198149 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:14.198125 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a09ac6d-1d69-4d01-8863-5e693b5533df-maas-api-tls\") pod \"maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd\" (UID: \"9a09ac6d-1d69-4d01-8863-5e693b5533df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd" Nov 02 23:59:14.400114 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:14.400085 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd" Nov 02 23:59:14.816317 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:14.816176 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd"] Nov 02 23:59:15.377475 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:15.377432 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd" event={"ID":"9a09ac6d-1d69-4d01-8863-5e693b5533df","Type":"ContainerStarted","Data":"ec59433575410c360a62e13b88a061ee64307e9b750f7ef051600a8361f7c416"} Nov 02 23:59:16.386392 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:16.386350 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd" event={"ID":"9a09ac6d-1d69-4d01-8863-5e693b5533df","Type":"ContainerStarted","Data":"dffc1321fa9688964f2ec7762389a36e1eaae8ecb2e4654bcedb89fbe92775a1"} Nov 02 23:59:16.386832 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:16.386440 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd" Nov 02 23:59:16.508481 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:16.508428 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd" podStartSLOduration=2.55242505 podStartE2EDuration="3.508413719s" totalImagesPullingTime="955.988669ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:59:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:59:16.506037142 +0000 UTC m=+4005.474568930" watchObservedRunningTime="2026-11-02 23:59:16.508413719 +0000 UTC m=+4005.476945503" Nov 02 23:59:16.689968 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:16.689153 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj"] Nov 02 23:59:16.689968 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:16.689477 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" podUID="56fa477d-d2c0-4a6b-a3b4-51cdf5420002" containerName="istio-proxy" containerID="cri-o://8416e217afdbe07c8c6c0566892e65f356cdf325120eb0bf69ca9170b4a2d521" gracePeriod=30 Nov 02 23:59:21.965496 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:21.965462 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" Nov 02 23:59:22.150252 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.150216 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-istiod-ca-cert\") pod \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\" (UID: \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\") " Nov 02 23:59:22.150424 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.150265 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-workload-socket\") pod \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\" (UID: \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\") " Nov 02 23:59:22.150424 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.150308 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-istio-data\") pod \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\" (UID: \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\") " Nov 02 23:59:22.150424 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.150339 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-istio-podinfo\") pod \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\" (UID: \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\") " Nov 02 23:59:22.150424 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.150391 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-kube-api-access-2jv79\" (UniqueName: \"kubernetes.io/projected/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-kube-api-access-2jv79\") pod \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\" (UID: \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\") " Nov 02 23:59:22.150706 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.150422 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-credential-socket\") pod \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\" (UID: \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\") " Nov 02 23:59:22.150706 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.150518 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-istio-envoy\") pod \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\" (UID: \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\") " Nov 02 23:59:22.150706 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.150578 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-workload-certs\") pod \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\" (UID: \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\") " Nov 02 23:59:22.150706 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.150656 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-istio-token\" (UniqueName: \"kubernetes.io/projected/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-istio-token\") pod \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\" (UID: \"56fa477d-d2c0-4a6b-a3b4-51cdf5420002\") " Nov 02 23:59:22.150706 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.150669 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-istio-data" pod "56fa477d-d2c0-4a6b-a3b4-51cdf5420002" (UID: "56fa477d-d2c0-4a6b-a3b4-51cdf5420002"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:59:22.150706 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.150684 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-workload-socket" pod "56fa477d-d2c0-4a6b-a3b4-51cdf5420002" (UID: "56fa477d-d2c0-4a6b-a3b4-51cdf5420002"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:59:22.151061 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.150714 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-istiod-ca-cert" pod "56fa477d-d2c0-4a6b-a3b4-51cdf5420002" (UID: "56fa477d-d2c0-4a6b-a3b4-51cdf5420002"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:59:22.151061 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.150917 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-workload-certs" pod "56fa477d-d2c0-4a6b-a3b4-51cdf5420002" (UID: "56fa477d-d2c0-4a6b-a3b4-51cdf5420002"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:59:22.151061 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.151023 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-credential-socket" pod "56fa477d-d2c0-4a6b-a3b4-51cdf5420002" (UID: "56fa477d-d2c0-4a6b-a3b4-51cdf5420002"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:59:22.151369 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.151197 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:59:22.151369 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.151222 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:59:22.151369 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.151236 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:59:22.151369 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.151271 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:59:22.151369 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.151288 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:59:22.153045 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.153017 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-istio-envoy" pod "56fa477d-d2c0-4a6b-a3b4-51cdf5420002" (UID: "56fa477d-d2c0-4a6b-a3b4-51cdf5420002"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:59:22.153393 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.153371 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-kube-api-access-2jv79" pod "56fa477d-d2c0-4a6b-a3b4-51cdf5420002" (UID: "56fa477d-d2c0-4a6b-a3b4-51cdf5420002"). InnerVolumeSpecName "kube-api-access-2jv79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:59:22.153452 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.153406 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-istio-token" pod "56fa477d-d2c0-4a6b-a3b4-51cdf5420002" (UID: "56fa477d-d2c0-4a6b-a3b4-51cdf5420002"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:59:22.153497 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.153490 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-istio-podinfo" pod "56fa477d-d2c0-4a6b-a3b4-51cdf5420002" (UID: "56fa477d-d2c0-4a6b-a3b4-51cdf5420002"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:59:22.252397 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.252355 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:59:22.252397 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.252387 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:59:22.252397 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.252399 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jv79\" (UniqueName: \"kubernetes.io/projected/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-kube-api-access-2jv79\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:59:22.252699 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.252410 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56fa477d-d2c0-4a6b-a3b4-51cdf5420002-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:59:22.400342 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.400311 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd" Nov 02 23:59:22.448768 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.448695 5076 generic.go:356] "Generic (PLEG): container finished" podID="56fa477d-d2c0-4a6b-a3b4-51cdf5420002" containerID="8416e217afdbe07c8c6c0566892e65f356cdf325120eb0bf69ca9170b4a2d521" exitCode=0 Nov 02 23:59:22.448996 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.448973 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" event={"ID":"56fa477d-d2c0-4a6b-a3b4-51cdf5420002","Type":"ContainerDied","Data":"8416e217afdbe07c8c6c0566892e65f356cdf325120eb0bf69ca9170b4a2d521"} Nov 02 23:59:22.449058 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.449033 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" event={"ID":"56fa477d-d2c0-4a6b-a3b4-51cdf5420002","Type":"ContainerDied","Data":"9bfc5fc56bcd34bc61a370e3adfccc4f9d144e3ad5932994831fce33bc05cfde"} Nov 02 23:59:22.449100 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.449061 5076 scope.go:122] "RemoveContainer" containerID="8416e217afdbe07c8c6c0566892e65f356cdf325120eb0bf69ca9170b4a2d521" Nov 02 23:59:22.449248 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.449231 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj" Nov 02 23:59:22.462067 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.462044 5076 scope.go:122] "RemoveContainer" containerID="1727703d39998e9704197b736d617afd642a267537407ef2852dd65c691ee0d3" Nov 02 23:59:22.472308 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.472292 5076 scope.go:122] "RemoveContainer" containerID="8416e217afdbe07c8c6c0566892e65f356cdf325120eb0bf69ca9170b4a2d521" Nov 02 23:59:22.472599 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:59:22.472578 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8416e217afdbe07c8c6c0566892e65f356cdf325120eb0bf69ca9170b4a2d521\": container with ID starting with 8416e217afdbe07c8c6c0566892e65f356cdf325120eb0bf69ca9170b4a2d521 not found: ID does not exist" containerID="8416e217afdbe07c8c6c0566892e65f356cdf325120eb0bf69ca9170b4a2d521" Nov 02 23:59:22.472649 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.472608 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8416e217afdbe07c8c6c0566892e65f356cdf325120eb0bf69ca9170b4a2d521"} err="failed to get container status \"8416e217afdbe07c8c6c0566892e65f356cdf325120eb0bf69ca9170b4a2d521\": rpc error: code = NotFound desc = could not find container \"8416e217afdbe07c8c6c0566892e65f356cdf325120eb0bf69ca9170b4a2d521\": container with ID starting with 8416e217afdbe07c8c6c0566892e65f356cdf325120eb0bf69ca9170b4a2d521 not found: ID does not exist" Nov 02 23:59:22.472649 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.472625 5076 scope.go:122] "RemoveContainer" containerID="1727703d39998e9704197b736d617afd642a267537407ef2852dd65c691ee0d3" Nov 02 23:59:22.472880 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:59:22.472863 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1727703d39998e9704197b736d617afd642a267537407ef2852dd65c691ee0d3\": container with ID starting with 1727703d39998e9704197b736d617afd642a267537407ef2852dd65c691ee0d3 not found: ID does not exist" containerID="1727703d39998e9704197b736d617afd642a267537407ef2852dd65c691ee0d3" Nov 02 23:59:22.473021 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.472885 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1727703d39998e9704197b736d617afd642a267537407ef2852dd65c691ee0d3"} err="failed to get container status \"1727703d39998e9704197b736d617afd642a267537407ef2852dd65c691ee0d3\": rpc error: code = NotFound desc = could not find container \"1727703d39998e9704197b736d617afd642a267537407ef2852dd65c691ee0d3\": container with ID starting with 1727703d39998e9704197b736d617afd642a267537407ef2852dd65c691ee0d3 not found: ID does not exist" Nov 02 23:59:22.624642 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.624562 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj"] Nov 02 23:59:22.630270 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:22.630240 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-a8a1dd-openshift-default-57bc447c69-xdcpj"] Nov 02 23:59:23.150228 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:23.150193 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56fa477d-d2c0-4a6b-a3b4-51cdf5420002" path="/var/lib/kubelet/pods/56fa477d-d2c0-4a6b-a3b4-51cdf5420002/volumes" Nov 02 23:59:25.469933 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:25.469869 5076 generic.go:356] "Generic (PLEG): container finished" podID="22d4e172-af11-4e35-9363-c4959295db10" containerID="a10016e2a412e13601247f08a90dbc56969d729dbac9493a49d0f680607503e5" exitCode=0 Nov 02 23:59:25.470496 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:25.469936 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" event={"ID":"22d4e172-af11-4e35-9363-c4959295db10","Type":"ContainerDied","Data":"a10016e2a412e13601247f08a90dbc56969d729dbac9493a49d0f680607503e5"} Nov 02 23:59:25.470567 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:25.470490 5076 scope.go:122] "RemoveContainer" containerID="a10016e2a412e13601247f08a90dbc56969d729dbac9493a49d0f680607503e5" Nov 02 23:59:25.472565 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:25.472530 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:59:25.472676 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:25.472623 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:59:25.472739 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:25.472672 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:59:26.478626 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:26.478589 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" event={"ID":"22d4e172-af11-4e35-9363-c4959295db10","Type":"ContainerStarted","Data":"6d5f2c90b11688f1c8959485513c54d031dced945b42cdf1971fe30222b88ac4"} Nov 02 23:59:26.896118 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:26.896031 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" Nov 02 23:59:26.896323 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:26.896303 5076 patch_prober.go:29] interesting pod/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 02 23:59:26.896374 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:26.896348 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" podUID="22d4e172-af11-4e35-9363-c4959295db10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 02 23:59:27.896555 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:27.896523 5076 patch_prober.go:29] interesting pod/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 02 23:59:27.896985 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:27.896574 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" podUID="22d4e172-af11-4e35-9363-c4959295db10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 02 23:59:28.896966 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:28.896930 5076 patch_prober.go:29] interesting pod/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 02 23:59:28.897367 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:28.896981 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" podUID="22d4e172-af11-4e35-9363-c4959295db10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 02 23:59:29.896598 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:29.896566 5076 patch_prober.go:29] interesting pod/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 02 23:59:29.896785 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:29.896621 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" podUID="22d4e172-af11-4e35-9363-c4959295db10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 02 23:59:30.896159 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:30.896122 5076 patch_prober.go:29] interesting pod/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 02 23:59:30.896541 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:30.896171 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" podUID="22d4e172-af11-4e35-9363-c4959295db10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 02 23:59:31.896517 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:31.896489 5076 patch_prober.go:29] interesting pod/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 02 23:59:31.897045 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:31.896966 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" podUID="22d4e172-af11-4e35-9363-c4959295db10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 02 23:59:32.901277 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:32.901192 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" Nov 02 23:59:32.901717 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:32.901660 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" Nov 02 23:59:32.902350 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:32.902328 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" Nov 02 23:59:39.053068 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:39.053028 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx"] Nov 02 23:59:39.053879 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:39.053858 5076 cpu_manager.go:414] "containerMap: removing container" podUID="56fa477d-d2c0-4a6b-a3b4-51cdf5420002" containerName="istio-proxy" Nov 02 23:59:39.053952 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:39.053883 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56fa477d-d2c0-4a6b-a3b4-51cdf5420002" containerName="istio-proxy" Nov 02 23:59:39.053952 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:39.053936 5076 cpu_manager.go:414] "containerMap: removing container" podUID="56fa477d-d2c0-4a6b-a3b4-51cdf5420002" containerName="istio-proxy" Nov 02 23:59:39.053952 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:39.053944 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56fa477d-d2c0-4a6b-a3b4-51cdf5420002" containerName="istio-proxy" Nov 02 23:59:39.054159 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:39.054146 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="56fa477d-d2c0-4a6b-a3b4-51cdf5420002" containerName="istio-proxy" Nov 02 23:59:39.054198 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:39.054173 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="56fa477d-d2c0-4a6b-a3b4-51cdf5420002" containerName="istio-proxy" Nov 02 23:59:39.057862 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:39.057844 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" Nov 02 23:59:39.159663 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:39.159627 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx"] Nov 02 23:59:39.229956 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:39.229918 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/905534e2-1b25-484c-9030-0185f3148cec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-89gnx\" (UID: \"905534e2-1b25-484c-9030-0185f3148cec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" Nov 02 23:59:39.230131 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:39.230112 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/905534e2-1b25-484c-9030-0185f3148cec-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-89gnx\" (UID: \"905534e2-1b25-484c-9030-0185f3148cec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" Nov 02 23:59:39.230184 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:39.230149 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/905534e2-1b25-484c-9030-0185f3148cec-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-89gnx\" (UID: \"905534e2-1b25-484c-9030-0185f3148cec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" Nov 02 23:59:39.230255 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:39.230243 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/905534e2-1b25-484c-9030-0185f3148cec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-89gnx\" (UID: \"905534e2-1b25-484c-9030-0185f3148cec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" Nov 02 23:59:39.230294 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:39.230272 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/905534e2-1b25-484c-9030-0185f3148cec-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-89gnx\" (UID: \"905534e2-1b25-484c-9030-0185f3148cec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" Nov 02 23:59:39.230294 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:39.230290 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/905534e2-1b25-484c-9030-0185f3148cec-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-89gnx\" (UID: \"905534e2-1b25-484c-9030-0185f3148cec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" Nov 02 23:59:39.230428 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:39.230408 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/905534e2-1b25-484c-9030-0185f3148cec-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-89gnx\" (UID: \"905534e2-1b25-484c-9030-0185f3148cec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" Nov 02 23:59:39.230468 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:39.230439 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l56l\" (UniqueName: \"kubernetes.io/projected/905534e2-1b25-484c-9030-0185f3148cec-kube-api-access-5l56l\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-89gnx\" (UID: \"905534e2-1b25-484c-9030-0185f3148cec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" Nov 02 23:59:39.230468 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:39.230460 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/905534e2-1b25-484c-9030-0185f3148cec-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-89gnx\" (UID: \"905534e2-1b25-484c-9030-0185f3148cec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" Nov 02 23:59:39.331334 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:39.331251 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/905534e2-1b25-484c-9030-0185f3148cec-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-89gnx\" (UID: \"905534e2-1b25-484c-9030-0185f3148cec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" Nov 02 23:59:39.331334 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:39.331289 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l56l\" (UniqueName: \"kubernetes.io/projected/905534e2-1b25-484c-9030-0185f3148cec-kube-api-access-5l56l\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-89gnx\" (UID: \"905534e2-1b25-484c-9030-0185f3148cec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" Nov 02 23:59:39.331334 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:39.331313 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/905534e2-1b25-484c-9030-0185f3148cec-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-89gnx\" (UID: \"905534e2-1b25-484c-9030-0185f3148cec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" Nov 02 23:59:39.331334 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:39.331332 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/905534e2-1b25-484c-9030-0185f3148cec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-89gnx\" (UID: \"905534e2-1b25-484c-9030-0185f3148cec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" Nov 02 23:59:39.331638 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:39.331455 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/905534e2-1b25-484c-9030-0185f3148cec-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-89gnx\" (UID: \"905534e2-1b25-484c-9030-0185f3148cec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" Nov 02 23:59:39.331638 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:39.331482 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/905534e2-1b25-484c-9030-0185f3148cec-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-89gnx\" (UID: \"905534e2-1b25-484c-9030-0185f3148cec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" Nov 02 23:59:39.331638 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:39.331525 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/905534e2-1b25-484c-9030-0185f3148cec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-89gnx\" (UID: \"905534e2-1b25-484c-9030-0185f3148cec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" Nov 02 23:59:39.331638 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:39.331562 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/905534e2-1b25-484c-9030-0185f3148cec-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-89gnx\" (UID: \"905534e2-1b25-484c-9030-0185f3148cec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" Nov 02 23:59:39.331638 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:39.331585 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/905534e2-1b25-484c-9030-0185f3148cec-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-89gnx\" (UID: \"905534e2-1b25-484c-9030-0185f3148cec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" Nov 02 23:59:39.331970 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:39.331715 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/905534e2-1b25-484c-9030-0185f3148cec-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-89gnx\" (UID: \"905534e2-1b25-484c-9030-0185f3148cec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" Nov 02 23:59:39.331970 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:39.331784 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/905534e2-1b25-484c-9030-0185f3148cec-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-89gnx\" (UID: \"905534e2-1b25-484c-9030-0185f3148cec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" Nov 02 23:59:39.332114 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:39.332094 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/905534e2-1b25-484c-9030-0185f3148cec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-89gnx\" (UID: \"905534e2-1b25-484c-9030-0185f3148cec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" Nov 02 23:59:39.332160 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:39.332104 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/905534e2-1b25-484c-9030-0185f3148cec-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-89gnx\" (UID: \"905534e2-1b25-484c-9030-0185f3148cec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" Nov 02 23:59:39.332160 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:39.332144 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/905534e2-1b25-484c-9030-0185f3148cec-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-89gnx\" (UID: \"905534e2-1b25-484c-9030-0185f3148cec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" Nov 02 23:59:39.333903 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:39.333868 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/905534e2-1b25-484c-9030-0185f3148cec-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-89gnx\" (UID: \"905534e2-1b25-484c-9030-0185f3148cec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" Nov 02 23:59:39.334074 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:39.334056 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/905534e2-1b25-484c-9030-0185f3148cec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-89gnx\" (UID: \"905534e2-1b25-484c-9030-0185f3148cec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" Nov 02 23:59:39.385974 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:39.385946 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/905534e2-1b25-484c-9030-0185f3148cec-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-89gnx\" (UID: \"905534e2-1b25-484c-9030-0185f3148cec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" Nov 02 23:59:39.386179 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:39.386157 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l56l\" (UniqueName: \"kubernetes.io/projected/905534e2-1b25-484c-9030-0185f3148cec-kube-api-access-5l56l\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-89gnx\" (UID: \"905534e2-1b25-484c-9030-0185f3148cec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" Nov 02 23:59:39.671757 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:39.671674 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" Nov 02 23:59:39.884841 ip-10-0-130-211 kubenswrapper[5076]: E1102 23:59:39.884799 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 02 23:59:40.117915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:40.109579 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx"] Nov 02 23:59:40.148546 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:40.143708 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:59:40.148546 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:40.143795 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:59:40.148546 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:40.143840 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:59:40.565801 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:40.565760 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" event={"ID":"905534e2-1b25-484c-9030-0185f3148cec","Type":"ContainerStarted","Data":"28c264b6df8b858f92ca502b53065b4f85ed636dcb35fd51846eeee5ccf7e7dc"} Nov 02 23:59:40.565998 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:40.565818 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" event={"ID":"905534e2-1b25-484c-9030-0185f3148cec","Type":"ContainerStarted","Data":"6e93fa98d475eea1dfbc955df4c7607bca17c60b6e224934c4336e090a7bc9e4"} Nov 02 23:59:40.672578 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:40.672547 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" Nov 02 23:59:40.682795 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:40.682736 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" podStartSLOduration=1.6827238709999999 podStartE2EDuration="1.682723871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:59:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:59:40.679183334 +0000 UTC m=+4029.647715147" watchObservedRunningTime="2026-11-02 23:59:40.682723871 +0000 UTC m=+4029.651255655" Nov 02 23:59:41.673823 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:41.673782 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-89gnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 02 23:59:41.674265 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:41.673840 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" podUID="905534e2-1b25-484c-9030-0185f3148cec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 02 23:59:42.672679 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:42.672636 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-89gnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 02 23:59:42.672871 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:42.672694 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" podUID="905534e2-1b25-484c-9030-0185f3148cec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 02 23:59:42.739160 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:42.739126 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv"] Nov 02 23:59:42.743470 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:42.743442 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" Nov 02 23:59:42.794540 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:42.794507 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bae87c4b-16e4-4fb3-a5db-cc529edebd50-istio-data\") pod \"e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv\" (UID: \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\") " pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" Nov 02 23:59:42.795419 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:42.795392 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fp9l\" (UniqueName: \"kubernetes.io/projected/bae87c4b-16e4-4fb3-a5db-cc529edebd50-kube-api-access-2fp9l\") pod \"e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv\" (UID: \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\") " pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" Nov 02 23:59:42.795635 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:42.795619 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae87c4b-16e4-4fb3-a5db-cc529edebd50-workload-socket\") pod \"e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv\" (UID: \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\") " pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" Nov 02 23:59:42.795725 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:42.795652 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bae87c4b-16e4-4fb3-a5db-cc529edebd50-istiod-ca-cert\") pod \"e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv\" (UID: \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\") " pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" Nov 02 23:59:42.795725 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:42.795689 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bae87c4b-16e4-4fb3-a5db-cc529edebd50-istio-token\") pod \"e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv\" (UID: \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\") " pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" Nov 02 23:59:42.795837 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:42.795736 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bae87c4b-16e4-4fb3-a5db-cc529edebd50-istio-envoy\") pod \"e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv\" (UID: \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\") " pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" Nov 02 23:59:42.795837 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:42.795818 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bae87c4b-16e4-4fb3-a5db-cc529edebd50-workload-certs\") pod \"e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv\" (UID: \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\") " pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" Nov 02 23:59:42.796037 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:42.795851 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae87c4b-16e4-4fb3-a5db-cc529edebd50-credential-socket\") pod \"e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv\" (UID: \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\") " pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" Nov 02 23:59:42.796037 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:42.795929 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bae87c4b-16e4-4fb3-a5db-cc529edebd50-istio-podinfo\") pod \"e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv\" (UID: \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\") " pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" Nov 02 23:59:42.799576 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:42.798531 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv"] Nov 02 23:59:42.855256 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:42.855221 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1f29e6-openshift-default-dockercfg-wwm2h\"" Nov 02 23:59:42.958914 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:42.950802 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bae87c4b-16e4-4fb3-a5db-cc529edebd50-istio-data\") pod \"e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv\" (UID: \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\") " pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" Nov 02 23:59:42.958914 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:42.950915 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fp9l\" (UniqueName: \"kubernetes.io/projected/bae87c4b-16e4-4fb3-a5db-cc529edebd50-kube-api-access-2fp9l\") pod \"e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv\" (UID: \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\") " pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" Nov 02 23:59:42.958914 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:42.950960 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae87c4b-16e4-4fb3-a5db-cc529edebd50-workload-socket\") pod \"e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv\" (UID: \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\") " pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" Nov 02 23:59:42.958914 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:42.950988 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bae87c4b-16e4-4fb3-a5db-cc529edebd50-istiod-ca-cert\") pod \"e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv\" (UID: \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\") " pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" Nov 02 23:59:42.958914 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:42.951020 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bae87c4b-16e4-4fb3-a5db-cc529edebd50-istio-token\") pod \"e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv\" (UID: \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\") " pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" Nov 02 23:59:42.958914 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:42.951065 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bae87c4b-16e4-4fb3-a5db-cc529edebd50-istio-envoy\") pod \"e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv\" (UID: \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\") " pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" Nov 02 23:59:42.958914 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:42.951154 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bae87c4b-16e4-4fb3-a5db-cc529edebd50-workload-certs\") pod \"e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv\" (UID: \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\") " pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" Nov 02 23:59:42.958914 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:42.951226 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae87c4b-16e4-4fb3-a5db-cc529edebd50-credential-socket\") pod \"e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv\" (UID: \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\") " pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" Nov 02 23:59:42.958914 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:42.951291 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bae87c4b-16e4-4fb3-a5db-cc529edebd50-istio-podinfo\") pod \"e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv\" (UID: \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\") " pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" Nov 02 23:59:42.958914 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:42.952528 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bae87c4b-16e4-4fb3-a5db-cc529edebd50-istio-data\") pod \"e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv\" (UID: \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\") " pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" Nov 02 23:59:42.958914 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:42.953746 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae87c4b-16e4-4fb3-a5db-cc529edebd50-workload-socket\") pod \"e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv\" (UID: \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\") " pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" Nov 02 23:59:42.958914 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:42.958575 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bae87c4b-16e4-4fb3-a5db-cc529edebd50-workload-certs\") pod \"e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv\" (UID: \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\") " pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" Nov 02 23:59:42.958914 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:42.958819 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae87c4b-16e4-4fb3-a5db-cc529edebd50-credential-socket\") pod \"e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv\" (UID: \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\") " pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" Nov 02 23:59:42.999828 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:42.978879 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bae87c4b-16e4-4fb3-a5db-cc529edebd50-istiod-ca-cert\") pod \"e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv\" (UID: \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\") " pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" Nov 02 23:59:43.006916 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:43.006864 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bae87c4b-16e4-4fb3-a5db-cc529edebd50-istio-podinfo\") pod \"e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv\" (UID: \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\") " pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" Nov 02 23:59:43.040777 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:43.013166 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bae87c4b-16e4-4fb3-a5db-cc529edebd50-istio-envoy\") pod \"e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv\" (UID: \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\") " pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" Nov 02 23:59:43.043331 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:43.043247 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fp9l\" (UniqueName: \"kubernetes.io/projected/bae87c4b-16e4-4fb3-a5db-cc529edebd50-kube-api-access-2fp9l\") pod \"e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv\" (UID: \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\") " pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" Nov 02 23:59:43.058839 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:43.057413 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bae87c4b-16e4-4fb3-a5db-cc529edebd50-istio-token\") pod \"e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv\" (UID: \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\") " pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" Nov 02 23:59:43.058839 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:43.057642 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" Nov 02 23:59:43.539528 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:59:43.539487 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae87c4b_16e4_4fb3_a5db_cc529edebd50.slice/crio-4eeecfbe6b2c7ea9005907b41d56bb2dad5d44d6a81bf660d74d2260e4f602fe WatchSource:0}: Error finding container 4eeecfbe6b2c7ea9005907b41d56bb2dad5d44d6a81bf660d74d2260e4f602fe: Status 404 returned error can't find the container with id 4eeecfbe6b2c7ea9005907b41d56bb2dad5d44d6a81bf660d74d2260e4f602fe Nov 02 23:59:43.539671 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:43.539556 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv"] Nov 02 23:59:43.542249 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:43.542216 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:59:43.542384 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:43.542292 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:59:43.542384 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:43.542336 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:59:43.638437 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:43.638397 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" event={"ID":"bae87c4b-16e4-4fb3-a5db-cc529edebd50","Type":"ContainerStarted","Data":"46545615b2a8604f569e61ab2c699867b0207a23d61de70cf124c6bb7155aa6f"} Nov 02 23:59:43.638692 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:43.638453 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" event={"ID":"bae87c4b-16e4-4fb3-a5db-cc529edebd50","Type":"ContainerStarted","Data":"4eeecfbe6b2c7ea9005907b41d56bb2dad5d44d6a81bf660d74d2260e4f602fe"} Nov 02 23:59:43.672249 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:43.672213 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-89gnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 02 23:59:43.672433 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:43.672271 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" podUID="905534e2-1b25-484c-9030-0185f3148cec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 02 23:59:43.750191 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:43.750128 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" podStartSLOduration=1.750109695 podStartE2EDuration="1.750109695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:59:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:59:43.748184994 +0000 UTC m=+4032.716716778" watchObservedRunningTime="2026-11-02 23:59:43.750109695 +0000 UTC m=+4032.718641481" Nov 02 23:59:44.062350 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:44.062318 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" Nov 02 23:59:44.063837 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:44.063809 5076 patch_prober.go:29] interesting pod/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 02 23:59:44.063995 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:44.063854 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" podUID="bae87c4b-16e4-4fb3-a5db-cc529edebd50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 02 23:59:44.672958 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:44.672925 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-89gnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 02 23:59:44.673147 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:44.672978 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" podUID="905534e2-1b25-484c-9030-0185f3148cec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 02 23:59:45.067640 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:45.067610 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" Nov 02 23:59:45.650583 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:45.650552 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" Nov 02 23:59:45.651713 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:45.651693 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" Nov 02 23:59:45.672079 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:45.672052 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-89gnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 02 23:59:45.672214 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:45.672107 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" podUID="905534e2-1b25-484c-9030-0185f3148cec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 02 23:59:46.672791 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:46.672757 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-89gnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 02 23:59:46.673308 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:46.672814 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" podUID="905534e2-1b25-484c-9030-0185f3148cec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 02 23:59:47.672207 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:47.672171 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-89gnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 02 23:59:47.672387 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:47.672242 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" podUID="905534e2-1b25-484c-9030-0185f3148cec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 02 23:59:49.046152 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:49.046110 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" Nov 02 23:59:49.046861 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:49.046617 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" Nov 02 23:59:49.048915 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:49.048861 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" Nov 02 23:59:54.173305 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:54.173270 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh"] Nov 02 23:59:54.178687 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:54.178663 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" Nov 02 23:59:54.271692 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:54.271659 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh"] Nov 02 23:59:54.280180 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:54.280149 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c9698d7-4d4b-44e6-8165-106b31bcae2b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-66nkh\" (UID: \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" Nov 02 23:59:54.280358 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:54.280221 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c9698d7-4d4b-44e6-8165-106b31bcae2b-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-66nkh\" (UID: \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" Nov 02 23:59:54.280358 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:54.280300 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9698d7-4d4b-44e6-8165-106b31bcae2b-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-66nkh\" (UID: \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" Nov 02 23:59:54.280358 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:54.280347 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9698d7-4d4b-44e6-8165-106b31bcae2b-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-66nkh\" (UID: \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" Nov 02 23:59:54.280543 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:54.280435 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c9698d7-4d4b-44e6-8165-106b31bcae2b-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-66nkh\" (UID: \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" Nov 02 23:59:54.280543 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:54.280512 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c9698d7-4d4b-44e6-8165-106b31bcae2b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-66nkh\" (UID: \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" Nov 02 23:59:54.280691 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:54.280667 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c9698d7-4d4b-44e6-8165-106b31bcae2b-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-66nkh\" (UID: \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" Nov 02 23:59:54.280769 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:54.280714 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llmjc\" (UniqueName: \"kubernetes.io/projected/5c9698d7-4d4b-44e6-8165-106b31bcae2b-kube-api-access-llmjc\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-66nkh\" (UID: \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" Nov 02 23:59:54.280905 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:54.280871 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c9698d7-4d4b-44e6-8165-106b31bcae2b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-66nkh\" (UID: \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" Nov 02 23:59:54.381693 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:54.381657 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c9698d7-4d4b-44e6-8165-106b31bcae2b-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-66nkh\" (UID: \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" Nov 02 23:59:54.381693 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:54.381699 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9698d7-4d4b-44e6-8165-106b31bcae2b-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-66nkh\" (UID: \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" Nov 02 23:59:54.381994 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:54.381726 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9698d7-4d4b-44e6-8165-106b31bcae2b-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-66nkh\" (UID: \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" Nov 02 23:59:54.381994 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:54.381754 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c9698d7-4d4b-44e6-8165-106b31bcae2b-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-66nkh\" (UID: \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" Nov 02 23:59:54.381994 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:54.381776 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c9698d7-4d4b-44e6-8165-106b31bcae2b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-66nkh\" (UID: \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" Nov 02 23:59:54.381994 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:54.381852 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c9698d7-4d4b-44e6-8165-106b31bcae2b-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-66nkh\" (UID: \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" Nov 02 23:59:54.381994 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:54.381874 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llmjc\" (UniqueName: \"kubernetes.io/projected/5c9698d7-4d4b-44e6-8165-106b31bcae2b-kube-api-access-llmjc\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-66nkh\" (UID: \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" Nov 02 23:59:54.381994 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:54.381996 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c9698d7-4d4b-44e6-8165-106b31bcae2b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-66nkh\" (UID: \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" Nov 02 23:59:54.382319 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:54.382029 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c9698d7-4d4b-44e6-8165-106b31bcae2b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-66nkh\" (UID: \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" Nov 02 23:59:54.382319 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:54.382146 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9698d7-4d4b-44e6-8165-106b31bcae2b-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-66nkh\" (UID: \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" Nov 02 23:59:54.382319 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:54.382165 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c9698d7-4d4b-44e6-8165-106b31bcae2b-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-66nkh\" (UID: \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" Nov 02 23:59:54.382319 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:54.382234 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9698d7-4d4b-44e6-8165-106b31bcae2b-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-66nkh\" (UID: \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" Nov 02 23:59:54.382520 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:54.382424 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c9698d7-4d4b-44e6-8165-106b31bcae2b-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-66nkh\" (UID: \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" Nov 02 23:59:54.382744 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:54.382725 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c9698d7-4d4b-44e6-8165-106b31bcae2b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-66nkh\" (UID: \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" Nov 02 23:59:54.384159 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:54.384137 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c9698d7-4d4b-44e6-8165-106b31bcae2b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-66nkh\" (UID: \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" Nov 02 23:59:54.384380 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:54.384362 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c9698d7-4d4b-44e6-8165-106b31bcae2b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-66nkh\" (UID: \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" Nov 02 23:59:54.436075 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:54.436040 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c9698d7-4d4b-44e6-8165-106b31bcae2b-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-66nkh\" (UID: \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" Nov 02 23:59:54.436204 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:54.436157 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llmjc\" (UniqueName: \"kubernetes.io/projected/5c9698d7-4d4b-44e6-8165-106b31bcae2b-kube-api-access-llmjc\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-66nkh\" (UID: \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" Nov 02 23:59:54.495374 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:54.495337 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" Nov 02 23:59:54.917949 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:54.917915 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh"] Nov 02 23:59:54.918771 ip-10-0-130-211 kubenswrapper[5076]: W1102 23:59:54.918744 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c9698d7_4d4b_44e6_8165_106b31bcae2b.slice/crio-d971080f0d99064aae83ced42e09ca4cd9ea6b78d65bc93cf77b6327069fe38e WatchSource:0}: Error finding container d971080f0d99064aae83ced42e09ca4cd9ea6b78d65bc93cf77b6327069fe38e: Status 404 returned error can't find the container with id d971080f0d99064aae83ced42e09ca4cd9ea6b78d65bc93cf77b6327069fe38e Nov 02 23:59:54.922356 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:54.922321 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:59:54.922483 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:54.922403 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:59:54.922483 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:54.922442 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:59:55.777322 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:55.777271 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" event={"ID":"5c9698d7-4d4b-44e6-8165-106b31bcae2b","Type":"ContainerStarted","Data":"a95800e962c583503e5aa7bd4c2d76e57a10d386554236ab6b8eec952348c535"} Nov 02 23:59:55.777968 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:55.777940 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" event={"ID":"5c9698d7-4d4b-44e6-8165-106b31bcae2b","Type":"ContainerStarted","Data":"d971080f0d99064aae83ced42e09ca4cd9ea6b78d65bc93cf77b6327069fe38e"} Nov 02 23:59:55.933918 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:55.933210 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" podStartSLOduration=1.93319051 podStartE2EDuration="1.93319051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:59:55.931369055 +0000 UTC m=+4044.899900840" watchObservedRunningTime="2026-11-02 23:59:55.93319051 +0000 UTC m=+4044.901722296" Nov 02 23:59:56.498781 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:56.498744 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" Nov 02 23:59:56.499187 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:56.499161 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-66nkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Nov 02 23:59:56.499381 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:56.499355 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" podUID="5c9698d7-4d4b-44e6-8165-106b31bcae2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Nov 02 23:59:57.495720 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:57.495680 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-66nkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Nov 02 23:59:57.496174 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:57.495742 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" podUID="5c9698d7-4d4b-44e6-8165-106b31bcae2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Nov 02 23:59:58.496381 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:58.496343 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-66nkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Nov 02 23:59:58.496782 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:58.496412 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" podUID="5c9698d7-4d4b-44e6-8165-106b31bcae2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Nov 02 23:59:59.496166 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:59.496130 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-66nkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Nov 02 23:59:59.496353 ip-10-0-130-211 kubenswrapper[5076]: I1102 23:59:59.496187 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" podUID="5c9698d7-4d4b-44e6-8165-106b31bcae2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Nov 03 00:00:00.302403 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:00.302367 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894400-h9bgb"] Nov 03 00:00:00.332971 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:00.330962 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894400-h9bgb" Nov 03 00:00:00.399191 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:00.392006 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23493c5e-9c71-4ae2-ab48-4e7ff1f626bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894400-h9bgb\" (UID: \"23493c5e-9c71-4ae2-ab48-4e7ff1f626bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894400-h9bgb" Nov 03 00:00:00.399191 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:00.392184 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk9p7\" (UniqueName: \"kubernetes.io/projected/23493c5e-9c71-4ae2-ab48-4e7ff1f626bf-kube-api-access-pk9p7\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894400-h9bgb\" (UID: \"23493c5e-9c71-4ae2-ab48-4e7ff1f626bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894400-h9bgb" Nov 03 00:00:00.426916 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:00.408476 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894400-h9bgb"] Nov 03 00:00:00.439918 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:00.439392 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 03 00:00:00.493821 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:00.493736 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23493c5e-9c71-4ae2-ab48-4e7ff1f626bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894400-h9bgb\" (UID: \"23493c5e-9c71-4ae2-ab48-4e7ff1f626bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894400-h9bgb" Nov 03 00:00:00.494021 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:00.493969 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk9p7\" (UniqueName: \"kubernetes.io/projected/23493c5e-9c71-4ae2-ab48-4e7ff1f626bf-kube-api-access-pk9p7\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894400-h9bgb\" (UID: \"23493c5e-9c71-4ae2-ab48-4e7ff1f626bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894400-h9bgb" Nov 03 00:00:00.494595 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:00.494573 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23493c5e-9c71-4ae2-ab48-4e7ff1f626bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894400-h9bgb\" (UID: \"23493c5e-9c71-4ae2-ab48-4e7ff1f626bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894400-h9bgb" Nov 03 00:00:00.496259 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:00.496235 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-66nkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Nov 03 00:00:00.496307 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:00.496287 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" podUID="5c9698d7-4d4b-44e6-8165-106b31bcae2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Nov 03 00:00:00.549345 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:00.549306 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk9p7\" (UniqueName: \"kubernetes.io/projected/23493c5e-9c71-4ae2-ab48-4e7ff1f626bf-kube-api-access-pk9p7\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894400-h9bgb\" (UID: \"23493c5e-9c71-4ae2-ab48-4e7ff1f626bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894400-h9bgb" Nov 03 00:00:00.707389 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:00.707351 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894400-h9bgb" Nov 03 00:00:01.305230 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:01.303922 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894400-h9bgb"] Nov 03 00:00:01.496862 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:01.496817 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-66nkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Nov 03 00:00:01.497084 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:01.496885 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" podUID="5c9698d7-4d4b-44e6-8165-106b31bcae2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Nov 03 00:00:01.876695 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:01.876567 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894400-h9bgb" event={"ID":"23493c5e-9c71-4ae2-ab48-4e7ff1f626bf","Type":"ContainerStarted","Data":"a5f9c8b7470c1af19d94dce8bff01240a6ae98ad60351ecbc0866ebe6a95c3fd"} Nov 03 00:00:02.499938 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:02.497438 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-66nkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Nov 03 00:00:02.499938 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:02.497520 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" podUID="5c9698d7-4d4b-44e6-8165-106b31bcae2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Nov 03 00:00:03.007969 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:03.007056 5076 generic.go:356] "Generic (PLEG): container finished" podID="23493c5e-9c71-4ae2-ab48-4e7ff1f626bf" containerID="715a61c63c35fe06ce6fa7bfafe32d07760c2c9b62240260d99f838cda2c1f5c" exitCode=60 Nov 03 00:00:03.007969 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:03.007278 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894400-h9bgb" event={"ID":"23493c5e-9c71-4ae2-ab48-4e7ff1f626bf","Type":"ContainerDied","Data":"715a61c63c35fe06ce6fa7bfafe32d07760c2c9b62240260d99f838cda2c1f5c"} Nov 03 00:00:03.007969 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:03.007706 5076 scope.go:122] "RemoveContainer" containerID="715a61c63c35fe06ce6fa7bfafe32d07760c2c9b62240260d99f838cda2c1f5c" Nov 03 00:00:03.495929 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:03.495703 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-66nkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Nov 03 00:00:03.495929 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:03.495763 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" podUID="5c9698d7-4d4b-44e6-8165-106b31bcae2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Nov 03 00:00:04.023268 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:04.023174 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894400-h9bgb" event={"ID":"23493c5e-9c71-4ae2-ab48-4e7ff1f626bf","Type":"ContainerStarted","Data":"4ad82d68d43cd52423e589932b253cc076787f4f657b411770eac7a08134a7fa"} Nov 03 00:00:05.043942 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:05.043884 5076 generic.go:356] "Generic (PLEG): container finished" podID="23493c5e-9c71-4ae2-ab48-4e7ff1f626bf" containerID="4ad82d68d43cd52423e589932b253cc076787f4f657b411770eac7a08134a7fa" exitCode=60 Nov 03 00:00:05.044455 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:05.044077 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894400-h9bgb" event={"ID":"23493c5e-9c71-4ae2-ab48-4e7ff1f626bf","Type":"ContainerDied","Data":"4ad82d68d43cd52423e589932b253cc076787f4f657b411770eac7a08134a7fa"} Nov 03 00:00:05.044455 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:05.044146 5076 scope.go:122] "RemoveContainer" containerID="715a61c63c35fe06ce6fa7bfafe32d07760c2c9b62240260d99f838cda2c1f5c" Nov 03 00:00:05.044654 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:05.044632 5076 scope.go:122] "RemoveContainer" containerID="4ad82d68d43cd52423e589932b253cc076787f4f657b411770eac7a08134a7fa" Nov 03 00:00:05.045059 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:00:05.045028 5076 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894400-h9bgb_odh-ai-gateway-infra(23493c5e-9c71-4ae2-ab48-4e7ff1f626bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894400-h9bgb" podUID="23493c5e-9c71-4ae2-ab48-4e7ff1f626bf" Nov 03 00:00:05.097704 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:05.097665 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" Nov 03 00:00:05.098019 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:05.097993 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" Nov 03 00:00:05.098779 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:05.098759 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" Nov 03 00:00:06.055150 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:06.055124 5076 scope.go:122] "RemoveContainer" containerID="4ad82d68d43cd52423e589932b253cc076787f4f657b411770eac7a08134a7fa" Nov 03 00:00:06.055816 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:00:06.055787 5076 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894400-h9bgb_odh-ai-gateway-infra(23493c5e-9c71-4ae2-ab48-4e7ff1f626bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894400-h9bgb" podUID="23493c5e-9c71-4ae2-ab48-4e7ff1f626bf" Nov 03 00:00:09.883306 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:00:09.883266 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:00:11.101382 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:11.101345 5076 generic.go:356] "Generic (PLEG): container finished" podID="df94f65b-8ef5-4e2c-a749-93a97b9c68e0" containerID="d99611eee0218a1928e4a81cd7dc524b3fd2340685a3fefc42c6c1cd09bfc6bd" exitCode=0 Nov 03 00:00:11.101937 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:11.101397 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" event={"ID":"df94f65b-8ef5-4e2c-a749-93a97b9c68e0","Type":"ContainerDied","Data":"d99611eee0218a1928e4a81cd7dc524b3fd2340685a3fefc42c6c1cd09bfc6bd"} Nov 03 00:00:11.101937 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:11.101904 5076 scope.go:122] "RemoveContainer" containerID="d99611eee0218a1928e4a81cd7dc524b3fd2340685a3fefc42c6c1cd09bfc6bd" Nov 03 00:00:11.103599 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:11.103566 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:00:11.103700 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:11.103645 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:00:11.103700 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:11.103675 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:00:12.134818 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:12.134777 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" event={"ID":"df94f65b-8ef5-4e2c-a749-93a97b9c68e0","Type":"ContainerStarted","Data":"f123ac3e083c408b31f992d541505e785759db4952252c657c95422bd0862913"} Nov 03 00:00:12.718309 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:12.718273 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" Nov 03 00:00:12.734087 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:12.734050 5076 patch_prober.go:29] interesting pod/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Nov 03 00:00:12.734295 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:12.734112 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" podUID="df94f65b-8ef5-4e2c-a749-93a97b9c68e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Nov 03 00:00:13.728915 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:13.717290 5076 patch_prober.go:29] interesting pod/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Nov 03 00:00:13.728915 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:13.717356 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" podUID="df94f65b-8ef5-4e2c-a749-93a97b9c68e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Nov 03 00:00:14.727913 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:14.717307 5076 patch_prober.go:29] interesting pod/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Nov 03 00:00:14.727913 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:14.717386 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" podUID="df94f65b-8ef5-4e2c-a749-93a97b9c68e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Nov 03 00:00:15.141079 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:15.141005 5076 scope.go:122] "RemoveContainer" containerID="4ad82d68d43cd52423e589932b253cc076787f4f657b411770eac7a08134a7fa" Nov 03 00:00:15.729914 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:15.725312 5076 patch_prober.go:29] interesting pod/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Nov 03 00:00:15.729914 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:15.725374 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" podUID="df94f65b-8ef5-4e2c-a749-93a97b9c68e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Nov 03 00:00:16.166008 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:16.165973 5076 generic.go:356] "Generic (PLEG): container finished" podID="23493c5e-9c71-4ae2-ab48-4e7ff1f626bf" containerID="9f7383080c8971fe1d955e8cd2e3c5f37996a12872547ddf32413aef5f407652" exitCode=60 Nov 03 00:00:16.166466 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:16.166049 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894400-h9bgb" event={"ID":"23493c5e-9c71-4ae2-ab48-4e7ff1f626bf","Type":"ContainerDied","Data":"9f7383080c8971fe1d955e8cd2e3c5f37996a12872547ddf32413aef5f407652"} Nov 03 00:00:16.166466 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:16.166126 5076 scope.go:122] "RemoveContainer" containerID="4ad82d68d43cd52423e589932b253cc076787f4f657b411770eac7a08134a7fa" Nov 03 00:00:16.166563 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:16.166502 5076 scope.go:122] "RemoveContainer" containerID="9f7383080c8971fe1d955e8cd2e3c5f37996a12872547ddf32413aef5f407652" Nov 03 00:00:16.166855 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:00:16.166830 5076 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894400-h9bgb_odh-ai-gateway-infra(23493c5e-9c71-4ae2-ab48-4e7ff1f626bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894400-h9bgb" podUID="23493c5e-9c71-4ae2-ab48-4e7ff1f626bf" Nov 03 00:00:16.716748 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:16.716303 5076 patch_prober.go:29] interesting pod/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Nov 03 00:00:16.716748 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:16.716371 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" podUID="df94f65b-8ef5-4e2c-a749-93a97b9c68e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Nov 03 00:00:16.880366 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:16.880333 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" Nov 03 00:00:17.301666 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:17.301633 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894400-h9bgb"] Nov 03 00:00:17.475980 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:17.475948 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894400-h9bgb" Nov 03 00:00:17.649705 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:17.649616 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23493c5e-9c71-4ae2-ab48-4e7ff1f626bf-kube-api-access-pk9p7\" (UniqueName: \"kubernetes.io/projected/23493c5e-9c71-4ae2-ab48-4e7ff1f626bf-kube-api-access-pk9p7\") pod \"23493c5e-9c71-4ae2-ab48-4e7ff1f626bf\" (UID: \"23493c5e-9c71-4ae2-ab48-4e7ff1f626bf\") " Nov 03 00:00:17.649918 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:17.649710 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23493c5e-9c71-4ae2-ab48-4e7ff1f626bf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23493c5e-9c71-4ae2-ab48-4e7ff1f626bf-maas-api-service-ca\") pod \"23493c5e-9c71-4ae2-ab48-4e7ff1f626bf\" (UID: \"23493c5e-9c71-4ae2-ab48-4e7ff1f626bf\") " Nov 03 00:00:17.650141 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:17.650118 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23493c5e-9c71-4ae2-ab48-4e7ff1f626bf-maas-api-service-ca" pod "23493c5e-9c71-4ae2-ab48-4e7ff1f626bf" (UID: "23493c5e-9c71-4ae2-ab48-4e7ff1f626bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:00:17.651766 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:17.651736 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23493c5e-9c71-4ae2-ab48-4e7ff1f626bf-kube-api-access-pk9p7" pod "23493c5e-9c71-4ae2-ab48-4e7ff1f626bf" (UID: "23493c5e-9c71-4ae2-ab48-4e7ff1f626bf"). InnerVolumeSpecName "kube-api-access-pk9p7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:00:17.715934 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:17.715900 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" Nov 03 00:00:17.717034 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:17.717016 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" Nov 03 00:00:17.751435 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:17.751406 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk9p7\" (UniqueName: \"kubernetes.io/projected/23493c5e-9c71-4ae2-ab48-4e7ff1f626bf-kube-api-access-pk9p7\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:00:17.751435 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:17.751431 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23493c5e-9c71-4ae2-ab48-4e7ff1f626bf-maas-api-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:00:18.182274 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:18.182240 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894400-h9bgb" Nov 03 00:00:18.182491 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:18.182271 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894400-h9bgb" event={"ID":"23493c5e-9c71-4ae2-ab48-4e7ff1f626bf","Type":"ContainerDied","Data":"a5f9c8b7470c1af19d94dce8bff01240a6ae98ad60351ecbc0866ebe6a95c3fd"} Nov 03 00:00:18.182491 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:18.182339 5076 scope.go:122] "RemoveContainer" containerID="9f7383080c8971fe1d955e8cd2e3c5f37996a12872547ddf32413aef5f407652" Nov 03 00:00:18.339289 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:18.339255 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894400-h9bgb"] Nov 03 00:00:18.342491 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:18.342463 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894400-h9bgb"] Nov 03 00:00:19.149361 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:19.149326 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23493c5e-9c71-4ae2-ab48-4e7ff1f626bf" path="/var/lib/kubelet/pods/23493c5e-9c71-4ae2-ab48-4e7ff1f626bf/volumes" Nov 03 00:00:25.723257 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:25.723213 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf"] Nov 03 00:00:25.724101 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:25.724078 5076 cpu_manager.go:414] "containerMap: removing container" podUID="23493c5e-9c71-4ae2-ab48-4e7ff1f626bf" containerName="cleanup" Nov 03 00:00:25.724261 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:25.724102 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23493c5e-9c71-4ae2-ab48-4e7ff1f626bf" containerName="cleanup" Nov 03 00:00:25.724261 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:25.724130 5076 cpu_manager.go:414] "containerMap: removing container" podUID="23493c5e-9c71-4ae2-ab48-4e7ff1f626bf" containerName="cleanup" Nov 03 00:00:25.724261 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:25.724138 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23493c5e-9c71-4ae2-ab48-4e7ff1f626bf" containerName="cleanup" Nov 03 00:00:25.724261 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:25.724153 5076 cpu_manager.go:414] "containerMap: removing container" podUID="23493c5e-9c71-4ae2-ab48-4e7ff1f626bf" containerName="cleanup" Nov 03 00:00:25.724261 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:25.724160 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23493c5e-9c71-4ae2-ab48-4e7ff1f626bf" containerName="cleanup" Nov 03 00:00:25.724523 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:25.724376 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="23493c5e-9c71-4ae2-ab48-4e7ff1f626bf" containerName="cleanup" Nov 03 00:00:25.724523 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:25.724394 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="23493c5e-9c71-4ae2-ab48-4e7ff1f626bf" containerName="cleanup" Nov 03 00:00:25.728300 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:25.728277 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" Nov 03 00:00:25.782257 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:25.782224 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-666a38-openshift-default-dockercfg-xbjck\"" Nov 03 00:00:25.805215 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:25.805175 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0af00e3b-e17f-4ebe-b17b-740f4f185417-istio-token\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf\" (UID: \"0af00e3b-e17f-4ebe-b17b-740f4f185417\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" Nov 03 00:00:25.805399 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:25.805304 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0af00e3b-e17f-4ebe-b17b-740f4f185417-istiod-ca-cert\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf\" (UID: \"0af00e3b-e17f-4ebe-b17b-740f4f185417\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" Nov 03 00:00:25.805399 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:25.805366 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9qdq\" (UniqueName: \"kubernetes.io/projected/0af00e3b-e17f-4ebe-b17b-740f4f185417-kube-api-access-p9qdq\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf\" (UID: \"0af00e3b-e17f-4ebe-b17b-740f4f185417\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" Nov 03 00:00:25.805509 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:25.805399 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0af00e3b-e17f-4ebe-b17b-740f4f185417-istio-podinfo\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf\" (UID: \"0af00e3b-e17f-4ebe-b17b-740f4f185417\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" Nov 03 00:00:25.805509 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:25.805428 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0af00e3b-e17f-4ebe-b17b-740f4f185417-credential-socket\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf\" (UID: \"0af00e3b-e17f-4ebe-b17b-740f4f185417\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" Nov 03 00:00:25.805593 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:25.805561 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0af00e3b-e17f-4ebe-b17b-740f4f185417-istio-data\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf\" (UID: \"0af00e3b-e17f-4ebe-b17b-740f4f185417\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" Nov 03 00:00:25.805689 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:25.805670 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0af00e3b-e17f-4ebe-b17b-740f4f185417-workload-certs\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf\" (UID: \"0af00e3b-e17f-4ebe-b17b-740f4f185417\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" Nov 03 00:00:25.805766 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:25.805751 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0af00e3b-e17f-4ebe-b17b-740f4f185417-workload-socket\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf\" (UID: \"0af00e3b-e17f-4ebe-b17b-740f4f185417\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" Nov 03 00:00:25.805820 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:25.805805 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0af00e3b-e17f-4ebe-b17b-740f4f185417-istio-envoy\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf\" (UID: \"0af00e3b-e17f-4ebe-b17b-740f4f185417\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" Nov 03 00:00:25.838504 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:25.838469 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf"] Nov 03 00:00:25.907861 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:25.907807 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0af00e3b-e17f-4ebe-b17b-740f4f185417-istiod-ca-cert\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf\" (UID: \"0af00e3b-e17f-4ebe-b17b-740f4f185417\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" Nov 03 00:00:25.908078 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:25.907869 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9qdq\" (UniqueName: \"kubernetes.io/projected/0af00e3b-e17f-4ebe-b17b-740f4f185417-kube-api-access-p9qdq\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf\" (UID: \"0af00e3b-e17f-4ebe-b17b-740f4f185417\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" Nov 03 00:00:25.913608 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:25.913554 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0af00e3b-e17f-4ebe-b17b-740f4f185417-istiod-ca-cert\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf\" (UID: \"0af00e3b-e17f-4ebe-b17b-740f4f185417\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" Nov 03 00:00:25.923869 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:25.923825 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0af00e3b-e17f-4ebe-b17b-740f4f185417-istio-podinfo\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf\" (UID: \"0af00e3b-e17f-4ebe-b17b-740f4f185417\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" Nov 03 00:00:25.924164 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:25.924144 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0af00e3b-e17f-4ebe-b17b-740f4f185417-credential-socket\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf\" (UID: \"0af00e3b-e17f-4ebe-b17b-740f4f185417\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" Nov 03 00:00:25.924348 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:25.924309 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0af00e3b-e17f-4ebe-b17b-740f4f185417-istio-data\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf\" (UID: \"0af00e3b-e17f-4ebe-b17b-740f4f185417\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" Nov 03 00:00:25.924542 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:25.924528 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0af00e3b-e17f-4ebe-b17b-740f4f185417-workload-certs\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf\" (UID: \"0af00e3b-e17f-4ebe-b17b-740f4f185417\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" Nov 03 00:00:25.924691 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:25.924676 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0af00e3b-e17f-4ebe-b17b-740f4f185417-workload-socket\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf\" (UID: \"0af00e3b-e17f-4ebe-b17b-740f4f185417\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" Nov 03 00:00:25.924831 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:25.924819 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0af00e3b-e17f-4ebe-b17b-740f4f185417-istio-envoy\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf\" (UID: \"0af00e3b-e17f-4ebe-b17b-740f4f185417\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" Nov 03 00:00:25.925049 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:25.925031 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0af00e3b-e17f-4ebe-b17b-740f4f185417-istio-token\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf\" (UID: \"0af00e3b-e17f-4ebe-b17b-740f4f185417\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" Nov 03 00:00:25.925215 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:25.925138 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0af00e3b-e17f-4ebe-b17b-740f4f185417-credential-socket\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf\" (UID: \"0af00e3b-e17f-4ebe-b17b-740f4f185417\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" Nov 03 00:00:25.926345 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:25.925826 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0af00e3b-e17f-4ebe-b17b-740f4f185417-istio-data\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf\" (UID: \"0af00e3b-e17f-4ebe-b17b-740f4f185417\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" Nov 03 00:00:25.926345 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:25.926022 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0af00e3b-e17f-4ebe-b17b-740f4f185417-workload-socket\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf\" (UID: \"0af00e3b-e17f-4ebe-b17b-740f4f185417\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" Nov 03 00:00:25.926345 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:25.926282 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0af00e3b-e17f-4ebe-b17b-740f4f185417-workload-certs\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf\" (UID: \"0af00e3b-e17f-4ebe-b17b-740f4f185417\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" Nov 03 00:00:25.943575 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:25.943546 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0af00e3b-e17f-4ebe-b17b-740f4f185417-istio-podinfo\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf\" (UID: \"0af00e3b-e17f-4ebe-b17b-740f4f185417\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" Nov 03 00:00:25.944720 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:25.944694 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0af00e3b-e17f-4ebe-b17b-740f4f185417-istio-envoy\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf\" (UID: \"0af00e3b-e17f-4ebe-b17b-740f4f185417\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" Nov 03 00:00:25.968951 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:25.968884 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9qdq\" (UniqueName: \"kubernetes.io/projected/0af00e3b-e17f-4ebe-b17b-740f4f185417-kube-api-access-p9qdq\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf\" (UID: \"0af00e3b-e17f-4ebe-b17b-740f4f185417\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" Nov 03 00:00:25.986495 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:25.986420 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0af00e3b-e17f-4ebe-b17b-740f4f185417-istio-token\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf\" (UID: \"0af00e3b-e17f-4ebe-b17b-740f4f185417\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" Nov 03 00:00:26.041730 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:26.041698 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" Nov 03 00:00:26.458898 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:26.458862 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf"] Nov 03 00:00:26.459368 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:00:26.459338 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af00e3b_e17f_4ebe_b17b_740f4f185417.slice/crio-70791d38b32a91adec123b6d02d30b320e259338b54aea00d26c6579fcaffe2a WatchSource:0}: Error finding container 70791d38b32a91adec123b6d02d30b320e259338b54aea00d26c6579fcaffe2a: Status 404 returned error can't find the container with id 70791d38b32a91adec123b6d02d30b320e259338b54aea00d26c6579fcaffe2a Nov 03 00:00:26.466247 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:26.466209 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:00:26.466354 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:26.466278 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:00:26.466354 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:26.466308 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:00:27.250057 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:27.250024 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" event={"ID":"0af00e3b-e17f-4ebe-b17b-740f4f185417","Type":"ContainerStarted","Data":"5369931aad2f81b9b11e71153bacb8801036060ce3b812d0fee174ea2699da68"} Nov 03 00:00:27.250705 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:27.250075 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" event={"ID":"0af00e3b-e17f-4ebe-b17b-740f4f185417","Type":"ContainerStarted","Data":"70791d38b32a91adec123b6d02d30b320e259338b54aea00d26c6579fcaffe2a"} Nov 03 00:00:27.411548 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:27.411480 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" podStartSLOduration=2.4114657680000002 podStartE2EDuration="2.411465768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:27.409230092 +0000 UTC m=+4076.377761885" watchObservedRunningTime="2026-11-03 00:00:27.411465768 +0000 UTC m=+4076.379997622" Nov 03 00:00:28.042153 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:28.042114 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" Nov 03 00:00:28.048366 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:28.048336 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" Nov 03 00:00:28.257565 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:28.257530 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" Nov 03 00:00:28.258640 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:28.258615 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" Nov 03 00:00:38.382842 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:38.382795 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx"] Nov 03 00:00:38.384534 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:38.384507 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="23493c5e-9c71-4ae2-ab48-4e7ff1f626bf" containerName="cleanup" Nov 03 00:00:38.388289 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:38.388247 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx" Nov 03 00:00:38.436354 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:38.436325 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx"] Nov 03 00:00:38.480269 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:38.480238 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-68d89f-serving-cert\"" Nov 03 00:00:38.546218 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:38.546178 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88a21350-c1be-4ac7-9f65-87fc3c18c136-metrics-certs\") pod \"maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx\" (UID: \"88a21350-c1be-4ac7-9f65-87fc3c18c136\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx" Nov 03 00:00:38.546400 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:38.546278 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dggwn\" (UniqueName: \"kubernetes.io/projected/88a21350-c1be-4ac7-9f65-87fc3c18c136-kube-api-access-dggwn\") pod \"maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx\" (UID: \"88a21350-c1be-4ac7-9f65-87fc3c18c136\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx" Nov 03 00:00:38.546400 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:38.546311 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88a21350-c1be-4ac7-9f65-87fc3c18c136-maas-api-tls\") pod \"maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx\" (UID: \"88a21350-c1be-4ac7-9f65-87fc3c18c136\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx" Nov 03 00:00:38.546520 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:38.546422 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88a21350-c1be-4ac7-9f65-87fc3c18c136-openshift-service-ca\") pod \"maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx\" (UID: \"88a21350-c1be-4ac7-9f65-87fc3c18c136\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx" Nov 03 00:00:38.655974 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:38.655771 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dggwn\" (UniqueName: \"kubernetes.io/projected/88a21350-c1be-4ac7-9f65-87fc3c18c136-kube-api-access-dggwn\") pod \"maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx\" (UID: \"88a21350-c1be-4ac7-9f65-87fc3c18c136\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx" Nov 03 00:00:38.655974 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:38.655848 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88a21350-c1be-4ac7-9f65-87fc3c18c136-maas-api-tls\") pod \"maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx\" (UID: \"88a21350-c1be-4ac7-9f65-87fc3c18c136\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx" Nov 03 00:00:38.655974 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:38.655954 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88a21350-c1be-4ac7-9f65-87fc3c18c136-openshift-service-ca\") pod \"maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx\" (UID: \"88a21350-c1be-4ac7-9f65-87fc3c18c136\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx" Nov 03 00:00:38.656266 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:38.656126 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88a21350-c1be-4ac7-9f65-87fc3c18c136-metrics-certs\") pod \"maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx\" (UID: \"88a21350-c1be-4ac7-9f65-87fc3c18c136\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx" Nov 03 00:00:38.671910 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:00:38.664235 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-68d89f-serving-cert: secret "maas-api-e2e-inf-a-68d89f-serving-cert" not found Nov 03 00:00:38.671910 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:00:38.664338 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88a21350-c1be-4ac7-9f65-87fc3c18c136-maas-api-tls podName:88a21350-c1be-4ac7-9f65-87fc3c18c136 nodeName:}" failed. No retries permitted until 2026-11-03 00:00:39.164313401 +0000 UTC m=+4088.132845165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/88a21350-c1be-4ac7-9f65-87fc3c18c136-maas-api-tls") pod "maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx" (UID: "88a21350-c1be-4ac7-9f65-87fc3c18c136") : secret "maas-api-e2e-inf-a-68d89f-serving-cert" not found Nov 03 00:00:38.671910 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:38.665536 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88a21350-c1be-4ac7-9f65-87fc3c18c136-openshift-service-ca\") pod \"maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx\" (UID: \"88a21350-c1be-4ac7-9f65-87fc3c18c136\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx" Nov 03 00:00:38.682610 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:38.682562 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88a21350-c1be-4ac7-9f65-87fc3c18c136-metrics-certs\") pod \"maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx\" (UID: \"88a21350-c1be-4ac7-9f65-87fc3c18c136\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx" Nov 03 00:00:38.719125 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:38.719091 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dggwn\" (UniqueName: \"kubernetes.io/projected/88a21350-c1be-4ac7-9f65-87fc3c18c136-kube-api-access-dggwn\") pod \"maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx\" (UID: \"88a21350-c1be-4ac7-9f65-87fc3c18c136\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx" Nov 03 00:00:39.205989 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:39.204816 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88a21350-c1be-4ac7-9f65-87fc3c18c136-maas-api-tls\") pod \"maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx\" (UID: \"88a21350-c1be-4ac7-9f65-87fc3c18c136\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx" Nov 03 00:00:39.216506 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:39.216454 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88a21350-c1be-4ac7-9f65-87fc3c18c136-maas-api-tls\") pod \"maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx\" (UID: \"88a21350-c1be-4ac7-9f65-87fc3c18c136\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx" Nov 03 00:00:39.301769 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:39.301737 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx" Nov 03 00:00:39.668832 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:39.668804 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx"] Nov 03 00:00:39.669525 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:00:39.669498 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88a21350_c1be_4ac7_9f65_87fc3c18c136.slice/crio-16f6b5452352cccf85967bfc9a00602a4a4ac19779e38b99966b9d9b73d50b82 WatchSource:0}: Error finding container 16f6b5452352cccf85967bfc9a00602a4a4ac19779e38b99966b9d9b73d50b82: Status 404 returned error can't find the container with id 16f6b5452352cccf85967bfc9a00602a4a4ac19779e38b99966b9d9b73d50b82 Nov 03 00:00:39.884807 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:00:39.884773 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:00:40.397972 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:40.397932 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx" event={"ID":"88a21350-c1be-4ac7-9f65-87fc3c18c136","Type":"ContainerStarted","Data":"16f6b5452352cccf85967bfc9a00602a4a4ac19779e38b99966b9d9b73d50b82"} Nov 03 00:00:41.410992 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:41.410151 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx" event={"ID":"88a21350-c1be-4ac7-9f65-87fc3c18c136","Type":"ContainerStarted","Data":"c5b6d556dcf9ee27153d5eafdda7d25769a04e8e80c6883f042a0fbbffc97a07"} Nov 03 00:00:41.410992 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:41.410396 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx" Nov 03 00:00:41.519823 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:41.519768 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx" podStartSLOduration=2.630439898 podStartE2EDuration="3.519753614s" totalImagesPullingTime="889.313716ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:41.519252238 +0000 UTC m=+4090.487784022" watchObservedRunningTime="2026-11-03 00:00:41.519753614 +0000 UTC m=+4090.488285398" Nov 03 00:00:47.422419 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:00:47.422389 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx" Nov 03 00:01:09.883750 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:01:09.883705 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:01:35.895231 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:35.895200 5076 generic.go:356] "Generic (PLEG): container finished" podID="bae87c4b-16e4-4fb3-a5db-cc529edebd50" containerID="46545615b2a8604f569e61ab2c699867b0207a23d61de70cf124c6bb7155aa6f" exitCode=0 Nov 03 00:01:35.895666 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:35.895267 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" event={"ID":"bae87c4b-16e4-4fb3-a5db-cc529edebd50","Type":"ContainerDied","Data":"46545615b2a8604f569e61ab2c699867b0207a23d61de70cf124c6bb7155aa6f"} Nov 03 00:01:35.895723 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:35.895684 5076 scope.go:122] "RemoveContainer" containerID="46545615b2a8604f569e61ab2c699867b0207a23d61de70cf124c6bb7155aa6f" Nov 03 00:01:35.897347 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:35.897321 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:01:35.897427 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:35.897384 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:01:35.897427 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:35.897418 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:01:35.909617 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:01:35.909588 5076 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae87c4b_16e4_4fb3_a5db_cc529edebd50.slice/crio-46545615b2a8604f569e61ab2c699867b0207a23d61de70cf124c6bb7155aa6f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae87c4b_16e4_4fb3_a5db_cc529edebd50.slice/crio-conmon-46545615b2a8604f569e61ab2c699867b0207a23d61de70cf124c6bb7155aa6f.scope\": RecentStats: unable to find data in memory cache]" Nov 03 00:01:36.903814 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:36.903778 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" event={"ID":"bae87c4b-16e4-4fb3-a5db-cc529edebd50","Type":"ContainerStarted","Data":"ab701962b8fc78e7e761b451d3aabd389a03420f7e187458cf10587c8380296f"} Nov 03 00:01:37.062810 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:37.062776 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" Nov 03 00:01:37.063122 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:37.063098 5076 patch_prober.go:29] interesting pod/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 03 00:01:37.063178 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:37.063146 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" podUID="bae87c4b-16e4-4fb3-a5db-cc529edebd50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 03 00:01:38.062334 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:38.062295 5076 patch_prober.go:29] interesting pod/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 03 00:01:38.062841 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:38.062360 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" podUID="bae87c4b-16e4-4fb3-a5db-cc529edebd50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 03 00:01:39.062951 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:39.062913 5076 patch_prober.go:29] interesting pod/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 03 00:01:39.063348 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:39.062965 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" podUID="bae87c4b-16e4-4fb3-a5db-cc529edebd50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 03 00:01:39.966057 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:01:39.966024 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:01:40.063044 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:40.063009 5076 patch_prober.go:29] interesting pod/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 03 00:01:40.063438 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:40.063074 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" podUID="bae87c4b-16e4-4fb3-a5db-cc529edebd50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 03 00:01:41.062257 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:41.062220 5076 patch_prober.go:29] interesting pod/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 03 00:01:41.062529 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:41.062298 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" podUID="bae87c4b-16e4-4fb3-a5db-cc529edebd50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 03 00:01:42.062859 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:42.062820 5076 patch_prober.go:29] interesting pod/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 03 00:01:42.063311 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:42.062933 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" podUID="bae87c4b-16e4-4fb3-a5db-cc529edebd50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 03 00:01:43.067933 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:43.067902 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" Nov 03 00:01:43.068436 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:43.068416 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" Nov 03 00:01:43.069209 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:43.069188 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" Nov 03 00:01:53.283289 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:53.283248 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87"] Nov 03 00:01:53.286750 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:53.286728 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87" Nov 03 00:01:53.342966 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:53.342926 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7917494a-587f-44da-b2f8-982a25c58526-maas-api-tls\") pod \"maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87\" (UID: \"7917494a-587f-44da-b2f8-982a25c58526\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87" Nov 03 00:01:53.343150 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:53.343048 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6sws\" (UniqueName: \"kubernetes.io/projected/7917494a-587f-44da-b2f8-982a25c58526-kube-api-access-w6sws\") pod \"maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87\" (UID: \"7917494a-587f-44da-b2f8-982a25c58526\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87" Nov 03 00:01:53.343150 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:53.343113 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7917494a-587f-44da-b2f8-982a25c58526-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87\" (UID: \"7917494a-587f-44da-b2f8-982a25c58526\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87" Nov 03 00:01:53.343150 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:53.343142 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7917494a-587f-44da-b2f8-982a25c58526-metrics-certs\") pod \"maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87\" (UID: \"7917494a-587f-44da-b2f8-982a25c58526\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87" Nov 03 00:01:53.384152 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:53.384117 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-666a38-serving-cert\"" Nov 03 00:01:53.389818 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:53.389782 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87"] Nov 03 00:01:53.444278 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:53.444244 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7917494a-587f-44da-b2f8-982a25c58526-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87\" (UID: \"7917494a-587f-44da-b2f8-982a25c58526\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87" Nov 03 00:01:53.444476 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:53.444292 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7917494a-587f-44da-b2f8-982a25c58526-metrics-certs\") pod \"maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87\" (UID: \"7917494a-587f-44da-b2f8-982a25c58526\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87" Nov 03 00:01:53.444552 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:53.444491 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7917494a-587f-44da-b2f8-982a25c58526-maas-api-tls\") pod \"maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87\" (UID: \"7917494a-587f-44da-b2f8-982a25c58526\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87" Nov 03 00:01:53.444646 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:53.444597 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6sws\" (UniqueName: \"kubernetes.io/projected/7917494a-587f-44da-b2f8-982a25c58526-kube-api-access-w6sws\") pod \"maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87\" (UID: \"7917494a-587f-44da-b2f8-982a25c58526\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87" Nov 03 00:01:53.444963 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:01:53.444939 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-666a38-serving-cert: secret "maas-api-e2e-extmod-b-666a38-serving-cert" not found Nov 03 00:01:53.445083 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:01:53.445019 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7917494a-587f-44da-b2f8-982a25c58526-maas-api-tls podName:7917494a-587f-44da-b2f8-982a25c58526 nodeName:}" failed. No retries permitted until 2026-11-03 00:01:53.94500067 +0000 UTC m=+4162.913532449 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7917494a-587f-44da-b2f8-982a25c58526-maas-api-tls") pod "maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87" (UID: "7917494a-587f-44da-b2f8-982a25c58526") : secret "maas-api-e2e-extmod-b-666a38-serving-cert" not found Nov 03 00:01:53.445170 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:53.445098 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7917494a-587f-44da-b2f8-982a25c58526-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87\" (UID: \"7917494a-587f-44da-b2f8-982a25c58526\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87" Nov 03 00:01:53.447125 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:53.447103 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7917494a-587f-44da-b2f8-982a25c58526-metrics-certs\") pod \"maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87\" (UID: \"7917494a-587f-44da-b2f8-982a25c58526\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87" Nov 03 00:01:53.499308 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:53.499273 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6sws\" (UniqueName: \"kubernetes.io/projected/7917494a-587f-44da-b2f8-982a25c58526-kube-api-access-w6sws\") pod \"maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87\" (UID: \"7917494a-587f-44da-b2f8-982a25c58526\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87" Nov 03 00:01:53.969496 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:53.969455 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7917494a-587f-44da-b2f8-982a25c58526-maas-api-tls\") pod \"maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87\" (UID: \"7917494a-587f-44da-b2f8-982a25c58526\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87" Nov 03 00:01:53.971990 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:53.971961 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7917494a-587f-44da-b2f8-982a25c58526-maas-api-tls\") pod \"maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87\" (UID: \"7917494a-587f-44da-b2f8-982a25c58526\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87" Nov 03 00:01:54.199132 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:54.199098 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87" Nov 03 00:01:54.567530 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:54.567487 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87"] Nov 03 00:01:54.568227 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:01:54.568200 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7917494a_587f_44da_b2f8_982a25c58526.slice/crio-5dfd1e8cf94044286d8924e1b579825eb696a8ac1f82a114a3e7fade9cd06cb9 WatchSource:0}: Error finding container 5dfd1e8cf94044286d8924e1b579825eb696a8ac1f82a114a3e7fade9cd06cb9: Status 404 returned error can't find the container with id 5dfd1e8cf94044286d8924e1b579825eb696a8ac1f82a114a3e7fade9cd06cb9 Nov 03 00:01:55.064366 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:55.064327 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87" event={"ID":"7917494a-587f-44da-b2f8-982a25c58526","Type":"ContainerStarted","Data":"5dfd1e8cf94044286d8924e1b579825eb696a8ac1f82a114a3e7fade9cd06cb9"} Nov 03 00:01:56.072653 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:56.072612 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87" event={"ID":"7917494a-587f-44da-b2f8-982a25c58526","Type":"ContainerStarted","Data":"df44a0ecd9ad6a87358c41dad096394f4b2f24cf71a8c7d387686bc87ace3e51"} Nov 03 00:01:56.073154 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:01:56.072705 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87" Nov 03 00:02:02.084602 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:02:02.084565 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87" Nov 03 00:02:02.196706 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:02:02.196650 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87" podStartSLOduration=8.213712453 podStartE2EDuration="9.196635374s" totalImagesPullingTime="982.922921ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:01:56.217867287 +0000 UTC m=+4165.186399072" watchObservedRunningTime="2026-11-03 00:02:02.196635374 +0000 UTC m=+4171.165167157" Nov 03 00:02:09.883511 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:02:09.883460 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:02:29.166475 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:02:29.165908 5076 patch_prober.go:29] interesting pod/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": read tcp 10.132.0.2:43538->10.132.0.195:15021: read: connection reset by peer" start-of-body= Nov 03 00:02:29.166475 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:02:29.165992 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" podUID="0af00e3b-e17f-4ebe-b17b-740f4f185417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": read tcp 10.132.0.2:43538->10.132.0.195:15021: read: connection reset by peer" Nov 03 00:02:29.711831 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:02:29.711798 5076 generic.go:356] "Generic (PLEG): container finished" podID="0af00e3b-e17f-4ebe-b17b-740f4f185417" containerID="5369931aad2f81b9b11e71153bacb8801036060ce3b812d0fee174ea2699da68" exitCode=0 Nov 03 00:02:29.712043 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:02:29.711870 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" event={"ID":"0af00e3b-e17f-4ebe-b17b-740f4f185417","Type":"ContainerDied","Data":"5369931aad2f81b9b11e71153bacb8801036060ce3b812d0fee174ea2699da68"} Nov 03 00:02:29.712331 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:02:29.712315 5076 scope.go:122] "RemoveContainer" containerID="5369931aad2f81b9b11e71153bacb8801036060ce3b812d0fee174ea2699da68" Nov 03 00:02:29.714130 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:02:29.714099 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:02:29.714227 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:02:29.714170 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:02:29.714227 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:02:29.714203 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:02:30.740420 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:02:30.740384 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" event={"ID":"0af00e3b-e17f-4ebe-b17b-740f4f185417","Type":"ContainerStarted","Data":"ed94b0c0bb72adc11dd8bea8bf8134a04b4bd003e0db3f9cbe4f01d0941c6a60"} Nov 03 00:02:31.042436 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:02:31.042350 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" Nov 03 00:02:31.042596 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:02:31.042528 5076 patch_prober.go:29] interesting pod/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 03 00:02:31.042596 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:02:31.042570 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" podUID="0af00e3b-e17f-4ebe-b17b-740f4f185417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 03 00:02:32.042115 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:02:32.042081 5076 patch_prober.go:29] interesting pod/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 03 00:02:32.042528 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:02:32.042131 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" podUID="0af00e3b-e17f-4ebe-b17b-740f4f185417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 03 00:02:32.065290 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:02:32.065253 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 03 00:02:32.153451 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:02:32.153423 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 03 00:02:33.042517 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:02:33.042474 5076 patch_prober.go:29] interesting pod/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 03 00:02:33.042968 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:02:33.042528 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" podUID="0af00e3b-e17f-4ebe-b17b-740f4f185417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 03 00:02:34.042480 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:02:34.042440 5076 patch_prober.go:29] interesting pod/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 03 00:02:34.042694 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:02:34.042501 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" podUID="0af00e3b-e17f-4ebe-b17b-740f4f185417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 03 00:02:35.043197 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:02:35.042870 5076 patch_prober.go:29] interesting pod/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 03 00:02:35.043197 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:02:35.042981 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" podUID="0af00e3b-e17f-4ebe-b17b-740f4f185417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 03 00:02:36.047848 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:02:36.047815 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" Nov 03 00:02:36.048276 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:02:36.048240 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" Nov 03 00:02:36.049225 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:02:36.049201 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" Nov 03 00:02:39.885844 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:02:39.885809 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:03:09.883998 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:03:09.883960 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:03:39.883582 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:03:39.883545 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:03:56.927697 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:03:56.927609 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh"] Nov 03 00:03:56.930868 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:03:56.930851 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh" Nov 03 00:03:56.981778 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:03:56.981722 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-6d61f9-serving-cert\"" Nov 03 00:03:57.031857 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:03:57.031824 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh"] Nov 03 00:03:57.039620 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:03:57.039594 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff048ceb-36d0-4d93-b323-aa8aa6e1c16a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh\" (UID: \"ff048ceb-36d0-4d93-b323-aa8aa6e1c16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh" Nov 03 00:03:57.039703 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:03:57.039644 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff048ceb-36d0-4d93-b323-aa8aa6e1c16a-metrics-certs\") pod \"maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh\" (UID: \"ff048ceb-36d0-4d93-b323-aa8aa6e1c16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh" Nov 03 00:03:57.039754 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:03:57.039735 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgdbm\" (UniqueName: \"kubernetes.io/projected/ff048ceb-36d0-4d93-b323-aa8aa6e1c16a-kube-api-access-lgdbm\") pod \"maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh\" (UID: \"ff048ceb-36d0-4d93-b323-aa8aa6e1c16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh" Nov 03 00:03:57.039808 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:03:57.039796 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff048ceb-36d0-4d93-b323-aa8aa6e1c16a-maas-api-tls\") pod \"maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh\" (UID: \"ff048ceb-36d0-4d93-b323-aa8aa6e1c16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh" Nov 03 00:03:57.141534 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:03:57.141500 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff048ceb-36d0-4d93-b323-aa8aa6e1c16a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh\" (UID: \"ff048ceb-36d0-4d93-b323-aa8aa6e1c16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh" Nov 03 00:03:57.141736 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:03:57.141558 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff048ceb-36d0-4d93-b323-aa8aa6e1c16a-metrics-certs\") pod \"maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh\" (UID: \"ff048ceb-36d0-4d93-b323-aa8aa6e1c16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh" Nov 03 00:03:57.141736 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:03:57.141603 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgdbm\" (UniqueName: \"kubernetes.io/projected/ff048ceb-36d0-4d93-b323-aa8aa6e1c16a-kube-api-access-lgdbm\") pod \"maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh\" (UID: \"ff048ceb-36d0-4d93-b323-aa8aa6e1c16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh" Nov 03 00:03:57.141736 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:03:57.141640 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff048ceb-36d0-4d93-b323-aa8aa6e1c16a-maas-api-tls\") pod \"maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh\" (UID: \"ff048ceb-36d0-4d93-b323-aa8aa6e1c16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh" Nov 03 00:03:57.141955 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:03:57.141760 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-6d61f9-serving-cert: secret "maas-api-e2e-inf-b-6d61f9-serving-cert" not found Nov 03 00:03:57.141955 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:03:57.141825 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff048ceb-36d0-4d93-b323-aa8aa6e1c16a-maas-api-tls podName:ff048ceb-36d0-4d93-b323-aa8aa6e1c16a nodeName:}" failed. No retries permitted until 2026-11-03 00:03:57.641804355 +0000 UTC m=+4286.610336134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ff048ceb-36d0-4d93-b323-aa8aa6e1c16a-maas-api-tls") pod "maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh" (UID: "ff048ceb-36d0-4d93-b323-aa8aa6e1c16a") : secret "maas-api-e2e-inf-b-6d61f9-serving-cert" not found Nov 03 00:03:57.142285 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:03:57.142265 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff048ceb-36d0-4d93-b323-aa8aa6e1c16a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh\" (UID: \"ff048ceb-36d0-4d93-b323-aa8aa6e1c16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh" Nov 03 00:03:57.144064 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:03:57.144041 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff048ceb-36d0-4d93-b323-aa8aa6e1c16a-metrics-certs\") pod \"maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh\" (UID: \"ff048ceb-36d0-4d93-b323-aa8aa6e1c16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh" Nov 03 00:03:57.234910 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:03:57.233689 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgdbm\" (UniqueName: \"kubernetes.io/projected/ff048ceb-36d0-4d93-b323-aa8aa6e1c16a-kube-api-access-lgdbm\") pod \"maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh\" (UID: \"ff048ceb-36d0-4d93-b323-aa8aa6e1c16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh" Nov 03 00:03:57.657008 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:03:57.656924 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff048ceb-36d0-4d93-b323-aa8aa6e1c16a-maas-api-tls\") pod \"maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh\" (UID: \"ff048ceb-36d0-4d93-b323-aa8aa6e1c16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh" Nov 03 00:03:57.659304 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:03:57.659279 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff048ceb-36d0-4d93-b323-aa8aa6e1c16a-maas-api-tls\") pod \"maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh\" (UID: \"ff048ceb-36d0-4d93-b323-aa8aa6e1c16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh" Nov 03 00:03:57.843446 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:03:57.843416 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh" Nov 03 00:03:58.216188 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:03:58.216156 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh"] Nov 03 00:03:58.217051 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:03:58.217025 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff048ceb_36d0_4d93_b323_aa8aa6e1c16a.slice/crio-afdb454ce08cc5670824a3aee3e70b6c4527542c9b09beed29d7f340f4386cc4 WatchSource:0}: Error finding container afdb454ce08cc5670824a3aee3e70b6c4527542c9b09beed29d7f340f4386cc4: Status 404 returned error can't find the container with id afdb454ce08cc5670824a3aee3e70b6c4527542c9b09beed29d7f340f4386cc4 Nov 03 00:03:58.408692 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:03:58.408661 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh" event={"ID":"ff048ceb-36d0-4d93-b323-aa8aa6e1c16a","Type":"ContainerStarted","Data":"afdb454ce08cc5670824a3aee3e70b6c4527542c9b09beed29d7f340f4386cc4"} Nov 03 00:03:59.418032 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:03:59.417995 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh" event={"ID":"ff048ceb-36d0-4d93-b323-aa8aa6e1c16a","Type":"ContainerStarted","Data":"0128d229f5e9a49ef71d9c6dc5e7084513a22d045012bbb52ec2a6c4fe3385bd"} Nov 03 00:03:59.418489 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:03:59.418077 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh" Nov 03 00:03:59.530592 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:03:59.530527 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh" podStartSLOduration=2.457132614 podStartE2EDuration="3.530504977s" totalImagesPullingTime="1.073372363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:03:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:03:59.528099654 +0000 UTC m=+4288.496631439" watchObservedRunningTime="2026-11-03 00:03:59.530504977 +0000 UTC m=+4288.499036762" Nov 03 00:04:05.429490 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:05.429452 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh" Nov 03 00:04:09.883998 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:04:09.883953 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:04:22.398314 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:22.398275 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv"] Nov 03 00:04:22.398724 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:22.398544 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" podUID="bae87c4b-16e4-4fb3-a5db-cc529edebd50" containerName="istio-proxy" containerID="cri-o://ab701962b8fc78e7e761b451d3aabd389a03420f7e187458cf10587c8380296f" gracePeriod=30 Nov 03 00:04:27.642769 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:27.642732 5076 generic.go:356] "Generic (PLEG): container finished" podID="bae87c4b-16e4-4fb3-a5db-cc529edebd50" containerID="ab701962b8fc78e7e761b451d3aabd389a03420f7e187458cf10587c8380296f" exitCode=0 Nov 03 00:04:27.643290 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:27.642950 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" event={"ID":"bae87c4b-16e4-4fb3-a5db-cc529edebd50","Type":"ContainerDied","Data":"ab701962b8fc78e7e761b451d3aabd389a03420f7e187458cf10587c8380296f"} Nov 03 00:04:27.643290 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:27.643011 5076 scope.go:122] "RemoveContainer" containerID="46545615b2a8604f569e61ab2c699867b0207a23d61de70cf124c6bb7155aa6f" Nov 03 00:04:27.672233 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:27.672206 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" Nov 03 00:04:27.850062 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:27.850028 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bae87c4b-16e4-4fb3-a5db-cc529edebd50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bae87c4b-16e4-4fb3-a5db-cc529edebd50-workload-certs\") pod \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\" (UID: \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\") " Nov 03 00:04:27.850271 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:27.850118 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bae87c4b-16e4-4fb3-a5db-cc529edebd50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bae87c4b-16e4-4fb3-a5db-cc529edebd50-istio-data\") pod \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\" (UID: \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\") " Nov 03 00:04:27.850271 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:27.850229 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bae87c4b-16e4-4fb3-a5db-cc529edebd50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bae87c4b-16e4-4fb3-a5db-cc529edebd50-istio-podinfo\") pod \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\" (UID: \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\") " Nov 03 00:04:27.850404 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:27.850274 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bae87c4b-16e4-4fb3-a5db-cc529edebd50-istio-token\" (UniqueName: \"kubernetes.io/projected/bae87c4b-16e4-4fb3-a5db-cc529edebd50-istio-token\") pod \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\" (UID: \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\") " Nov 03 00:04:27.850404 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:27.850381 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bae87c4b-16e4-4fb3-a5db-cc529edebd50-workload-certs" pod "bae87c4b-16e4-4fb3-a5db-cc529edebd50" (UID: "bae87c4b-16e4-4fb3-a5db-cc529edebd50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:04:27.850533 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:27.850410 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bae87c4b-16e4-4fb3-a5db-cc529edebd50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bae87c4b-16e4-4fb3-a5db-cc529edebd50-istio-envoy\") pod \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\" (UID: \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\") " Nov 03 00:04:27.850533 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:27.850421 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bae87c4b-16e4-4fb3-a5db-cc529edebd50-istio-data" pod "bae87c4b-16e4-4fb3-a5db-cc529edebd50" (UID: "bae87c4b-16e4-4fb3-a5db-cc529edebd50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:04:27.850533 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:27.850473 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bae87c4b-16e4-4fb3-a5db-cc529edebd50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae87c4b-16e4-4fb3-a5db-cc529edebd50-workload-socket\") pod \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\" (UID: \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\") " Nov 03 00:04:27.850533 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:27.850512 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bae87c4b-16e4-4fb3-a5db-cc529edebd50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae87c4b-16e4-4fb3-a5db-cc529edebd50-credential-socket\") pod \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\" (UID: \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\") " Nov 03 00:04:27.850748 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:27.850555 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bae87c4b-16e4-4fb3-a5db-cc529edebd50-kube-api-access-2fp9l\" (UniqueName: \"kubernetes.io/projected/bae87c4b-16e4-4fb3-a5db-cc529edebd50-kube-api-access-2fp9l\") pod \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\" (UID: \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\") " Nov 03 00:04:27.850748 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:27.850629 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bae87c4b-16e4-4fb3-a5db-cc529edebd50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bae87c4b-16e4-4fb3-a5db-cc529edebd50-istiod-ca-cert\") pod \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\" (UID: \"bae87c4b-16e4-4fb3-a5db-cc529edebd50\") " Nov 03 00:04:27.851379 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:27.851346 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bae87c4b-16e4-4fb3-a5db-cc529edebd50-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:04:27.851379 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:27.851375 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bae87c4b-16e4-4fb3-a5db-cc529edebd50-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:04:27.851595 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:27.851481 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bae87c4b-16e4-4fb3-a5db-cc529edebd50-credential-socket" pod "bae87c4b-16e4-4fb3-a5db-cc529edebd50" (UID: "bae87c4b-16e4-4fb3-a5db-cc529edebd50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:04:27.851683 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:27.851660 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bae87c4b-16e4-4fb3-a5db-cc529edebd50-istiod-ca-cert" pod "bae87c4b-16e4-4fb3-a5db-cc529edebd50" (UID: "bae87c4b-16e4-4fb3-a5db-cc529edebd50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:04:27.852629 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:27.852605 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bae87c4b-16e4-4fb3-a5db-cc529edebd50-workload-socket" pod "bae87c4b-16e4-4fb3-a5db-cc529edebd50" (UID: "bae87c4b-16e4-4fb3-a5db-cc529edebd50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:04:27.853397 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:27.853366 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bae87c4b-16e4-4fb3-a5db-cc529edebd50-kube-api-access-2fp9l" pod "bae87c4b-16e4-4fb3-a5db-cc529edebd50" (UID: "bae87c4b-16e4-4fb3-a5db-cc529edebd50"). InnerVolumeSpecName "kube-api-access-2fp9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:04:27.853506 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:27.853453 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bae87c4b-16e4-4fb3-a5db-cc529edebd50-istio-podinfo" pod "bae87c4b-16e4-4fb3-a5db-cc529edebd50" (UID: "bae87c4b-16e4-4fb3-a5db-cc529edebd50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:04:27.854776 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:27.854749 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bae87c4b-16e4-4fb3-a5db-cc529edebd50-istio-token" pod "bae87c4b-16e4-4fb3-a5db-cc529edebd50" (UID: "bae87c4b-16e4-4fb3-a5db-cc529edebd50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:04:27.854999 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:27.854972 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bae87c4b-16e4-4fb3-a5db-cc529edebd50-istio-envoy" pod "bae87c4b-16e4-4fb3-a5db-cc529edebd50" (UID: "bae87c4b-16e4-4fb3-a5db-cc529edebd50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:04:27.953286 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:27.953249 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bae87c4b-16e4-4fb3-a5db-cc529edebd50-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:04:27.953286 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:27.953285 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bae87c4b-16e4-4fb3-a5db-cc529edebd50-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:04:27.953501 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:27.953301 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bae87c4b-16e4-4fb3-a5db-cc529edebd50-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:04:27.953501 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:27.953312 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae87c4b-16e4-4fb3-a5db-cc529edebd50-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:04:27.953501 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:27.953321 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae87c4b-16e4-4fb3-a5db-cc529edebd50-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:04:27.953501 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:27.953329 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fp9l\" (UniqueName: \"kubernetes.io/projected/bae87c4b-16e4-4fb3-a5db-cc529edebd50-kube-api-access-2fp9l\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:04:27.953501 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:27.953339 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bae87c4b-16e4-4fb3-a5db-cc529edebd50-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:04:28.655942 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:28.655882 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" Nov 03 00:04:28.656360 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:28.655938 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv" event={"ID":"bae87c4b-16e4-4fb3-a5db-cc529edebd50","Type":"ContainerDied","Data":"4eeecfbe6b2c7ea9005907b41d56bb2dad5d44d6a81bf660d74d2260e4f602fe"} Nov 03 00:04:28.656360 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:28.656003 5076 scope.go:122] "RemoveContainer" containerID="ab701962b8fc78e7e761b451d3aabd389a03420f7e187458cf10587c8380296f" Nov 03 00:04:28.925045 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:28.924945 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv"] Nov 03 00:04:28.929155 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:28.929126 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1f29e6-openshift-default-67898645c-ksmkv"] Nov 03 00:04:29.149917 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:29.149870 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bae87c4b-16e4-4fb3-a5db-cc529edebd50" path="/var/lib/kubelet/pods/bae87c4b-16e4-4fb3-a5db-cc529edebd50/volumes" Nov 03 00:04:39.884724 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:04:39.884679 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:04:49.980845 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:49.980802 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg"] Nov 03 00:04:49.981377 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:49.981082 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg" podUID="3533e1bf-11a4-4c94-a310-6edeaa98a834" containerName="maas-api" containerID="cri-o://a675e8415013454b1ec975ce57e8e69599803b3f4b4e790ed2e4d5dae3741bfc" gracePeriod=30 Nov 03 00:04:50.326227 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:50.326200 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg" Nov 03 00:04:50.383275 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:50.383240 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3533e1bf-11a4-4c94-a310-6edeaa98a834-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3533e1bf-11a4-4c94-a310-6edeaa98a834-openshift-service-ca\") pod \"3533e1bf-11a4-4c94-a310-6edeaa98a834\" (UID: \"3533e1bf-11a4-4c94-a310-6edeaa98a834\") " Nov 03 00:04:50.383458 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:50.383299 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3533e1bf-11a4-4c94-a310-6edeaa98a834-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3533e1bf-11a4-4c94-a310-6edeaa98a834-metrics-certs\") pod \"3533e1bf-11a4-4c94-a310-6edeaa98a834\" (UID: \"3533e1bf-11a4-4c94-a310-6edeaa98a834\") " Nov 03 00:04:50.383458 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:50.383354 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3533e1bf-11a4-4c94-a310-6edeaa98a834-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3533e1bf-11a4-4c94-a310-6edeaa98a834-maas-api-tls\") pod \"3533e1bf-11a4-4c94-a310-6edeaa98a834\" (UID: \"3533e1bf-11a4-4c94-a310-6edeaa98a834\") " Nov 03 00:04:50.383561 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:50.383509 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3533e1bf-11a4-4c94-a310-6edeaa98a834-kube-api-access-fwbq2\" (UniqueName: \"kubernetes.io/projected/3533e1bf-11a4-4c94-a310-6edeaa98a834-kube-api-access-fwbq2\") pod \"3533e1bf-11a4-4c94-a310-6edeaa98a834\" (UID: \"3533e1bf-11a4-4c94-a310-6edeaa98a834\") " Nov 03 00:04:50.383706 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:50.383683 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3533e1bf-11a4-4c94-a310-6edeaa98a834-openshift-service-ca" pod "3533e1bf-11a4-4c94-a310-6edeaa98a834" (UID: "3533e1bf-11a4-4c94-a310-6edeaa98a834"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:04:50.384174 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:50.384144 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3533e1bf-11a4-4c94-a310-6edeaa98a834-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:04:50.385574 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:50.385546 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3533e1bf-11a4-4c94-a310-6edeaa98a834-metrics-certs" pod "3533e1bf-11a4-4c94-a310-6edeaa98a834" (UID: "3533e1bf-11a4-4c94-a310-6edeaa98a834"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:04:50.385574 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:50.385565 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3533e1bf-11a4-4c94-a310-6edeaa98a834-maas-api-tls" pod "3533e1bf-11a4-4c94-a310-6edeaa98a834" (UID: "3533e1bf-11a4-4c94-a310-6edeaa98a834"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:04:50.385760 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:50.385644 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3533e1bf-11a4-4c94-a310-6edeaa98a834-kube-api-access-fwbq2" pod "3533e1bf-11a4-4c94-a310-6edeaa98a834" (UID: "3533e1bf-11a4-4c94-a310-6edeaa98a834"). InnerVolumeSpecName "kube-api-access-fwbq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:04:50.488923 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:50.485908 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwbq2\" (UniqueName: \"kubernetes.io/projected/3533e1bf-11a4-4c94-a310-6edeaa98a834-kube-api-access-fwbq2\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:04:50.488923 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:50.485954 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3533e1bf-11a4-4c94-a310-6edeaa98a834-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:04:50.488923 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:50.485969 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3533e1bf-11a4-4c94-a310-6edeaa98a834-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:04:50.803922 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:50.803859 5076 generic.go:356] "Generic (PLEG): container finished" podID="3533e1bf-11a4-4c94-a310-6edeaa98a834" containerID="a675e8415013454b1ec975ce57e8e69599803b3f4b4e790ed2e4d5dae3741bfc" exitCode=0 Nov 03 00:04:50.804114 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:50.803920 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg" event={"ID":"3533e1bf-11a4-4c94-a310-6edeaa98a834","Type":"ContainerDied","Data":"a675e8415013454b1ec975ce57e8e69599803b3f4b4e790ed2e4d5dae3741bfc"} Nov 03 00:04:50.804114 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:50.803939 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg" Nov 03 00:04:50.804114 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:50.803979 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg" event={"ID":"3533e1bf-11a4-4c94-a310-6edeaa98a834","Type":"ContainerDied","Data":"8dc8d7deebae50390d4f1fa8880e889378201a0b96a6fb8a4eb16a09205ee77c"} Nov 03 00:04:50.804114 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:50.804005 5076 scope.go:122] "RemoveContainer" containerID="a675e8415013454b1ec975ce57e8e69599803b3f4b4e790ed2e4d5dae3741bfc" Nov 03 00:04:50.814616 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:50.814598 5076 scope.go:122] "RemoveContainer" containerID="a675e8415013454b1ec975ce57e8e69599803b3f4b4e790ed2e4d5dae3741bfc" Nov 03 00:04:50.814878 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:04:50.814862 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a675e8415013454b1ec975ce57e8e69599803b3f4b4e790ed2e4d5dae3741bfc\": container with ID starting with a675e8415013454b1ec975ce57e8e69599803b3f4b4e790ed2e4d5dae3741bfc not found: ID does not exist" containerID="a675e8415013454b1ec975ce57e8e69599803b3f4b4e790ed2e4d5dae3741bfc" Nov 03 00:04:50.814939 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:50.814901 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a675e8415013454b1ec975ce57e8e69599803b3f4b4e790ed2e4d5dae3741bfc"} err="failed to get container status \"a675e8415013454b1ec975ce57e8e69599803b3f4b4e790ed2e4d5dae3741bfc\": rpc error: code = NotFound desc = could not find container \"a675e8415013454b1ec975ce57e8e69599803b3f4b4e790ed2e4d5dae3741bfc\": container with ID starting with a675e8415013454b1ec975ce57e8e69599803b3f4b4e790ed2e4d5dae3741bfc not found: ID does not exist" Nov 03 00:04:50.963116 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:50.963080 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg"] Nov 03 00:04:51.009666 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:51.009630 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad18ae-766995d659-2rcwg"] Nov 03 00:04:51.219913 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:51.209279 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3533e1bf-11a4-4c94-a310-6edeaa98a834" path="/var/lib/kubelet/pods/3533e1bf-11a4-4c94-a310-6edeaa98a834/volumes" Nov 03 00:04:56.776327 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:56.776227 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp"] Nov 03 00:04:56.776861 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:04:56.776493 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" podUID="22d4e172-af11-4e35-9363-c4959295db10" containerName="istio-proxy" containerID="cri-o://6d5f2c90b11688f1c8959485513c54d031dced945b42cdf1971fe30222b88ac4" gracePeriod=30 Nov 03 00:05:02.066920 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.066877 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" Nov 03 00:05:02.142698 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.142663 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22d4e172-af11-4e35-9363-c4959295db10-istio-token\" (UniqueName: \"kubernetes.io/projected/22d4e172-af11-4e35-9363-c4959295db10-istio-token\") pod \"22d4e172-af11-4e35-9363-c4959295db10\" (UID: \"22d4e172-af11-4e35-9363-c4959295db10\") " Nov 03 00:05:02.142698 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.142704 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22d4e172-af11-4e35-9363-c4959295db10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d4e172-af11-4e35-9363-c4959295db10-credential-socket\") pod \"22d4e172-af11-4e35-9363-c4959295db10\" (UID: \"22d4e172-af11-4e35-9363-c4959295db10\") " Nov 03 00:05:02.143001 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.142749 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22d4e172-af11-4e35-9363-c4959295db10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22d4e172-af11-4e35-9363-c4959295db10-istio-data\") pod \"22d4e172-af11-4e35-9363-c4959295db10\" (UID: \"22d4e172-af11-4e35-9363-c4959295db10\") " Nov 03 00:05:02.143001 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.142838 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22d4e172-af11-4e35-9363-c4959295db10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22d4e172-af11-4e35-9363-c4959295db10-workload-certs\") pod \"22d4e172-af11-4e35-9363-c4959295db10\" (UID: \"22d4e172-af11-4e35-9363-c4959295db10\") " Nov 03 00:05:02.143001 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.142931 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22d4e172-af11-4e35-9363-c4959295db10-kube-api-access-tsx76\" (UniqueName: \"kubernetes.io/projected/22d4e172-af11-4e35-9363-c4959295db10-kube-api-access-tsx76\") pod \"22d4e172-af11-4e35-9363-c4959295db10\" (UID: \"22d4e172-af11-4e35-9363-c4959295db10\") " Nov 03 00:05:02.143001 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.142978 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22d4e172-af11-4e35-9363-c4959295db10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22d4e172-af11-4e35-9363-c4959295db10-istio-podinfo\") pod \"22d4e172-af11-4e35-9363-c4959295db10\" (UID: \"22d4e172-af11-4e35-9363-c4959295db10\") " Nov 03 00:05:02.143229 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.143006 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22d4e172-af11-4e35-9363-c4959295db10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22d4e172-af11-4e35-9363-c4959295db10-istiod-ca-cert\") pod \"22d4e172-af11-4e35-9363-c4959295db10\" (UID: \"22d4e172-af11-4e35-9363-c4959295db10\") " Nov 03 00:05:02.143229 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.143088 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22d4e172-af11-4e35-9363-c4959295db10-credential-socket" pod "22d4e172-af11-4e35-9363-c4959295db10" (UID: "22d4e172-af11-4e35-9363-c4959295db10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:05:02.143229 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.143148 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22d4e172-af11-4e35-9363-c4959295db10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d4e172-af11-4e35-9363-c4959295db10-workload-socket\") pod \"22d4e172-af11-4e35-9363-c4959295db10\" (UID: \"22d4e172-af11-4e35-9363-c4959295db10\") " Nov 03 00:05:02.143730 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.143368 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22d4e172-af11-4e35-9363-c4959295db10-istiod-ca-cert" pod "22d4e172-af11-4e35-9363-c4959295db10" (UID: "22d4e172-af11-4e35-9363-c4959295db10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:05:02.143730 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.143420 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22d4e172-af11-4e35-9363-c4959295db10-workload-socket" pod "22d4e172-af11-4e35-9363-c4959295db10" (UID: "22d4e172-af11-4e35-9363-c4959295db10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:05:02.143929 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.143760 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22d4e172-af11-4e35-9363-c4959295db10-istio-data" pod "22d4e172-af11-4e35-9363-c4959295db10" (UID: "22d4e172-af11-4e35-9363-c4959295db10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:05:02.143929 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.143824 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22d4e172-af11-4e35-9363-c4959295db10-workload-certs" pod "22d4e172-af11-4e35-9363-c4959295db10" (UID: "22d4e172-af11-4e35-9363-c4959295db10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:05:02.143929 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.143857 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22d4e172-af11-4e35-9363-c4959295db10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22d4e172-af11-4e35-9363-c4959295db10-istio-envoy\") pod \"22d4e172-af11-4e35-9363-c4959295db10\" (UID: \"22d4e172-af11-4e35-9363-c4959295db10\") " Nov 03 00:05:02.144601 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.144576 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22d4e172-af11-4e35-9363-c4959295db10-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:05:02.144715 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.144607 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d4e172-af11-4e35-9363-c4959295db10-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:05:02.144715 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.144623 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d4e172-af11-4e35-9363-c4959295db10-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:05:02.144715 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.144636 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22d4e172-af11-4e35-9363-c4959295db10-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:05:02.144715 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.144650 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22d4e172-af11-4e35-9363-c4959295db10-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:05:02.145817 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.145796 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22d4e172-af11-4e35-9363-c4959295db10-istio-token" pod "22d4e172-af11-4e35-9363-c4959295db10" (UID: "22d4e172-af11-4e35-9363-c4959295db10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:05:02.147263 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.147238 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22d4e172-af11-4e35-9363-c4959295db10-istio-podinfo" pod "22d4e172-af11-4e35-9363-c4959295db10" (UID: "22d4e172-af11-4e35-9363-c4959295db10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:05:02.147368 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.147244 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22d4e172-af11-4e35-9363-c4959295db10-istio-envoy" pod "22d4e172-af11-4e35-9363-c4959295db10" (UID: "22d4e172-af11-4e35-9363-c4959295db10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:05:02.148600 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.148569 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22d4e172-af11-4e35-9363-c4959295db10-kube-api-access-tsx76" pod "22d4e172-af11-4e35-9363-c4959295db10" (UID: "22d4e172-af11-4e35-9363-c4959295db10"). InnerVolumeSpecName "kube-api-access-tsx76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:05:02.246416 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.246380 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsx76\" (UniqueName: \"kubernetes.io/projected/22d4e172-af11-4e35-9363-c4959295db10-kube-api-access-tsx76\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:05:02.246416 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.246414 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22d4e172-af11-4e35-9363-c4959295db10-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:05:02.246416 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.246425 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22d4e172-af11-4e35-9363-c4959295db10-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:05:02.246732 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.246434 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22d4e172-af11-4e35-9363-c4959295db10-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:05:02.256794 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.256758 5076 generic.go:356] "Generic (PLEG): container finished" podID="22d4e172-af11-4e35-9363-c4959295db10" containerID="6d5f2c90b11688f1c8959485513c54d031dced945b42cdf1971fe30222b88ac4" exitCode=0 Nov 03 00:05:02.257032 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.256841 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" Nov 03 00:05:02.257032 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.256855 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" event={"ID":"22d4e172-af11-4e35-9363-c4959295db10","Type":"ContainerDied","Data":"6d5f2c90b11688f1c8959485513c54d031dced945b42cdf1971fe30222b88ac4"} Nov 03 00:05:02.257032 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.256955 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp" event={"ID":"22d4e172-af11-4e35-9363-c4959295db10","Type":"ContainerDied","Data":"0d8aeffde671ef9d4538322fc99c3dd0a5cfc2d48386dbe0b8702452840af46a"} Nov 03 00:05:02.257032 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.256981 5076 scope.go:122] "RemoveContainer" containerID="6d5f2c90b11688f1c8959485513c54d031dced945b42cdf1971fe30222b88ac4" Nov 03 00:05:02.269119 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.269093 5076 scope.go:122] "RemoveContainer" containerID="a10016e2a412e13601247f08a90dbc56969d729dbac9493a49d0f680607503e5" Nov 03 00:05:02.279467 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.279371 5076 scope.go:122] "RemoveContainer" containerID="6d5f2c90b11688f1c8959485513c54d031dced945b42cdf1971fe30222b88ac4" Nov 03 00:05:02.280025 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:05:02.279872 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d5f2c90b11688f1c8959485513c54d031dced945b42cdf1971fe30222b88ac4\": container with ID starting with 6d5f2c90b11688f1c8959485513c54d031dced945b42cdf1971fe30222b88ac4 not found: ID does not exist" containerID="6d5f2c90b11688f1c8959485513c54d031dced945b42cdf1971fe30222b88ac4" Nov 03 00:05:02.280025 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.279927 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d5f2c90b11688f1c8959485513c54d031dced945b42cdf1971fe30222b88ac4"} err="failed to get container status \"6d5f2c90b11688f1c8959485513c54d031dced945b42cdf1971fe30222b88ac4\": rpc error: code = NotFound desc = could not find container \"6d5f2c90b11688f1c8959485513c54d031dced945b42cdf1971fe30222b88ac4\": container with ID starting with 6d5f2c90b11688f1c8959485513c54d031dced945b42cdf1971fe30222b88ac4 not found: ID does not exist" Nov 03 00:05:02.280025 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.279955 5076 scope.go:122] "RemoveContainer" containerID="a10016e2a412e13601247f08a90dbc56969d729dbac9493a49d0f680607503e5" Nov 03 00:05:02.280498 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:05:02.280465 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a10016e2a412e13601247f08a90dbc56969d729dbac9493a49d0f680607503e5\": container with ID starting with a10016e2a412e13601247f08a90dbc56969d729dbac9493a49d0f680607503e5 not found: ID does not exist" containerID="a10016e2a412e13601247f08a90dbc56969d729dbac9493a49d0f680607503e5" Nov 03 00:05:02.280605 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.280498 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a10016e2a412e13601247f08a90dbc56969d729dbac9493a49d0f680607503e5"} err="failed to get container status \"a10016e2a412e13601247f08a90dbc56969d729dbac9493a49d0f680607503e5\": rpc error: code = NotFound desc = could not find container \"a10016e2a412e13601247f08a90dbc56969d729dbac9493a49d0f680607503e5\": container with ID starting with a10016e2a412e13601247f08a90dbc56969d729dbac9493a49d0f680607503e5 not found: ID does not exist" Nov 03 00:05:02.430389 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.430278 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp"] Nov 03 00:05:02.456731 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:02.456694 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad18ae-openshift-default-c9bdcb745-5scjp"] Nov 03 00:05:03.156965 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:03.156924 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22d4e172-af11-4e35-9363-c4959295db10" path="/var/lib/kubelet/pods/22d4e172-af11-4e35-9363-c4959295db10/volumes" Nov 03 00:05:07.316994 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:07.316952 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd"] Nov 03 00:05:07.317504 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:07.317215 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd" podUID="9a09ac6d-1d69-4d01-8863-5e693b5533df" containerName="maas-api" containerID="cri-o://dffc1321fa9688964f2ec7762389a36e1eaae8ecb2e4654bcedb89fbe92775a1" gracePeriod=30 Nov 03 00:05:07.395337 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:07.395287 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd" podUID="9a09ac6d-1d69-4d01-8863-5e693b5533df" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.190:8443/health\": dial tcp 10.132.0.190:8443: connect: connection refused" Nov 03 00:05:07.646780 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:07.646752 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd" Nov 03 00:05:07.723859 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:07.723820 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a09ac6d-1d69-4d01-8863-5e693b5533df-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a09ac6d-1d69-4d01-8863-5e693b5533df-metrics-certs\") pod \"9a09ac6d-1d69-4d01-8863-5e693b5533df\" (UID: \"9a09ac6d-1d69-4d01-8863-5e693b5533df\") " Nov 03 00:05:07.724103 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:07.723875 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a09ac6d-1d69-4d01-8863-5e693b5533df-kube-api-access-jhqb9\" (UniqueName: \"kubernetes.io/projected/9a09ac6d-1d69-4d01-8863-5e693b5533df-kube-api-access-jhqb9\") pod \"9a09ac6d-1d69-4d01-8863-5e693b5533df\" (UID: \"9a09ac6d-1d69-4d01-8863-5e693b5533df\") " Nov 03 00:05:07.724103 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:07.724007 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a09ac6d-1d69-4d01-8863-5e693b5533df-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a09ac6d-1d69-4d01-8863-5e693b5533df-openshift-service-ca\") pod \"9a09ac6d-1d69-4d01-8863-5e693b5533df\" (UID: \"9a09ac6d-1d69-4d01-8863-5e693b5533df\") " Nov 03 00:05:07.724103 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:07.724044 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a09ac6d-1d69-4d01-8863-5e693b5533df-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a09ac6d-1d69-4d01-8863-5e693b5533df-maas-api-tls\") pod \"9a09ac6d-1d69-4d01-8863-5e693b5533df\" (UID: \"9a09ac6d-1d69-4d01-8863-5e693b5533df\") " Nov 03 00:05:07.724430 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:07.724405 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a09ac6d-1d69-4d01-8863-5e693b5533df-openshift-service-ca" pod "9a09ac6d-1d69-4d01-8863-5e693b5533df" (UID: "9a09ac6d-1d69-4d01-8863-5e693b5533df"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:05:07.724551 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:07.724507 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a09ac6d-1d69-4d01-8863-5e693b5533df-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:05:07.726152 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:07.726128 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a09ac6d-1d69-4d01-8863-5e693b5533df-metrics-certs" pod "9a09ac6d-1d69-4d01-8863-5e693b5533df" (UID: "9a09ac6d-1d69-4d01-8863-5e693b5533df"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:05:07.726251 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:07.726175 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a09ac6d-1d69-4d01-8863-5e693b5533df-kube-api-access-jhqb9" pod "9a09ac6d-1d69-4d01-8863-5e693b5533df" (UID: "9a09ac6d-1d69-4d01-8863-5e693b5533df"). InnerVolumeSpecName "kube-api-access-jhqb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:05:07.726251 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:07.726218 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a09ac6d-1d69-4d01-8863-5e693b5533df-maas-api-tls" pod "9a09ac6d-1d69-4d01-8863-5e693b5533df" (UID: "9a09ac6d-1d69-4d01-8863-5e693b5533df"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:05:07.826610 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:07.826438 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a09ac6d-1d69-4d01-8863-5e693b5533df-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:05:07.826807 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:07.826621 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhqb9\" (UniqueName: \"kubernetes.io/projected/9a09ac6d-1d69-4d01-8863-5e693b5533df-kube-api-access-jhqb9\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:05:07.826807 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:07.826640 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a09ac6d-1d69-4d01-8863-5e693b5533df-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:05:08.351692 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:08.351652 5076 generic.go:356] "Generic (PLEG): container finished" podID="9a09ac6d-1d69-4d01-8863-5e693b5533df" containerID="dffc1321fa9688964f2ec7762389a36e1eaae8ecb2e4654bcedb89fbe92775a1" exitCode=0 Nov 03 00:05:08.352358 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:08.352332 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd" event={"ID":"9a09ac6d-1d69-4d01-8863-5e693b5533df","Type":"ContainerDied","Data":"dffc1321fa9688964f2ec7762389a36e1eaae8ecb2e4654bcedb89fbe92775a1"} Nov 03 00:05:08.352557 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:08.352539 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd" event={"ID":"9a09ac6d-1d69-4d01-8863-5e693b5533df","Type":"ContainerDied","Data":"ec59433575410c360a62e13b88a061ee64307e9b750f7ef051600a8361f7c416"} Nov 03 00:05:08.352689 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:08.352676 5076 scope.go:122] "RemoveContainer" containerID="dffc1321fa9688964f2ec7762389a36e1eaae8ecb2e4654bcedb89fbe92775a1" Nov 03 00:05:08.353164 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:08.353129 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd" Nov 03 00:05:08.389178 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:08.389149 5076 scope.go:122] "RemoveContainer" containerID="dffc1321fa9688964f2ec7762389a36e1eaae8ecb2e4654bcedb89fbe92775a1" Nov 03 00:05:08.389541 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:05:08.389503 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dffc1321fa9688964f2ec7762389a36e1eaae8ecb2e4654bcedb89fbe92775a1\": container with ID starting with dffc1321fa9688964f2ec7762389a36e1eaae8ecb2e4654bcedb89fbe92775a1 not found: ID does not exist" containerID="dffc1321fa9688964f2ec7762389a36e1eaae8ecb2e4654bcedb89fbe92775a1" Nov 03 00:05:08.389611 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:08.389554 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dffc1321fa9688964f2ec7762389a36e1eaae8ecb2e4654bcedb89fbe92775a1"} err="failed to get container status \"dffc1321fa9688964f2ec7762389a36e1eaae8ecb2e4654bcedb89fbe92775a1\": rpc error: code = NotFound desc = could not find container \"dffc1321fa9688964f2ec7762389a36e1eaae8ecb2e4654bcedb89fbe92775a1\": container with ID starting with dffc1321fa9688964f2ec7762389a36e1eaae8ecb2e4654bcedb89fbe92775a1 not found: ID does not exist" Nov 03 00:05:08.520692 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:08.520656 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd"] Nov 03 00:05:08.567863 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:08.567831 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e1be81-65b95dbccf-d2nnd"] Nov 03 00:05:09.148699 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:09.148665 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a09ac6d-1d69-4d01-8863-5e693b5533df" path="/var/lib/kubelet/pods/9a09ac6d-1d69-4d01-8863-5e693b5533df/volumes" Nov 03 00:05:09.884193 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:05:09.884157 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:05:17.816822 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:17.816336 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb"] Nov 03 00:05:17.816822 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:17.816566 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" podUID="df94f65b-8ef5-4e2c-a749-93a97b9c68e0" containerName="istio-proxy" containerID="cri-o://f123ac3e083c408b31f992d541505e785759db4952252c657c95422bd0862913" gracePeriod=30 Nov 03 00:05:23.108655 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.108589 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" Nov 03 00:05:23.251176 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.251133 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-workload-certs\") pod \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\" (UID: \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\") " Nov 03 00:05:23.251372 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.251196 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-kube-api-access-6c9mn\" (UniqueName: \"kubernetes.io/projected/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-kube-api-access-6c9mn\") pod \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\" (UID: \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\") " Nov 03 00:05:23.251372 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.251282 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-istio-data\") pod \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\" (UID: \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\") " Nov 03 00:05:23.251372 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.251339 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-istio-token\" (UniqueName: \"kubernetes.io/projected/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-istio-token\") pod \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\" (UID: \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\") " Nov 03 00:05:23.251572 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.251401 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-istiod-ca-cert\") pod \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\" (UID: \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\") " Nov 03 00:05:23.251572 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.251454 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-workload-certs" pod "df94f65b-8ef5-4e2c-a749-93a97b9c68e0" (UID: "df94f65b-8ef5-4e2c-a749-93a97b9c68e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:05:23.251572 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.251555 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-workload-socket\") pod \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\" (UID: \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\") " Nov 03 00:05:23.251748 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.251578 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-istio-data" pod "df94f65b-8ef5-4e2c-a749-93a97b9c68e0" (UID: "df94f65b-8ef5-4e2c-a749-93a97b9c68e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:05:23.251748 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.251591 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-istio-envoy\") pod \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\" (UID: \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\") " Nov 03 00:05:23.251748 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.251671 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-credential-socket\") pod \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\" (UID: \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\") " Nov 03 00:05:23.251748 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.251709 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-istio-podinfo\") pod \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\" (UID: \"df94f65b-8ef5-4e2c-a749-93a97b9c68e0\") " Nov 03 00:05:23.251995 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.251779 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-istiod-ca-cert" pod "df94f65b-8ef5-4e2c-a749-93a97b9c68e0" (UID: "df94f65b-8ef5-4e2c-a749-93a97b9c68e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:05:23.251995 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.251789 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-workload-socket" pod "df94f65b-8ef5-4e2c-a749-93a97b9c68e0" (UID: "df94f65b-8ef5-4e2c-a749-93a97b9c68e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:05:23.251995 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.251947 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-credential-socket" pod "df94f65b-8ef5-4e2c-a749-93a97b9c68e0" (UID: "df94f65b-8ef5-4e2c-a749-93a97b9c68e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:05:23.252294 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.252274 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:05:23.252369 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.252299 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:05:23.252369 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.252315 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:05:23.252369 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.252330 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:05:23.252369 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.252343 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:05:23.253725 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.253696 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-istio-envoy" pod "df94f65b-8ef5-4e2c-a749-93a97b9c68e0" (UID: "df94f65b-8ef5-4e2c-a749-93a97b9c68e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:05:23.253725 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.253717 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-istio-podinfo" pod "df94f65b-8ef5-4e2c-a749-93a97b9c68e0" (UID: "df94f65b-8ef5-4e2c-a749-93a97b9c68e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:05:23.253993 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.253970 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-kube-api-access-6c9mn" pod "df94f65b-8ef5-4e2c-a749-93a97b9c68e0" (UID: "df94f65b-8ef5-4e2c-a749-93a97b9c68e0"). InnerVolumeSpecName "kube-api-access-6c9mn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:05:23.254109 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.254051 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-istio-token" pod "df94f65b-8ef5-4e2c-a749-93a97b9c68e0" (UID: "df94f65b-8ef5-4e2c-a749-93a97b9c68e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:05:23.361071 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.360916 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:05:23.361071 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.360960 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:05:23.361071 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.360977 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:05:23.361071 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.360992 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c9mn\" (UniqueName: \"kubernetes.io/projected/df94f65b-8ef5-4e2c-a749-93a97b9c68e0-kube-api-access-6c9mn\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:05:23.529441 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.529404 5076 generic.go:356] "Generic (PLEG): container finished" podID="df94f65b-8ef5-4e2c-a749-93a97b9c68e0" containerID="f123ac3e083c408b31f992d541505e785759db4952252c657c95422bd0862913" exitCode=0 Nov 03 00:05:23.529617 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.529454 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" Nov 03 00:05:23.529617 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.529488 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" event={"ID":"df94f65b-8ef5-4e2c-a749-93a97b9c68e0","Type":"ContainerDied","Data":"f123ac3e083c408b31f992d541505e785759db4952252c657c95422bd0862913"} Nov 03 00:05:23.529617 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.529547 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb" event={"ID":"df94f65b-8ef5-4e2c-a749-93a97b9c68e0","Type":"ContainerDied","Data":"0843c2b2b2c9949259a6c559ab1b9e3e18fc8413afe3471c31b51510fde88804"} Nov 03 00:05:23.529617 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.529571 5076 scope.go:122] "RemoveContainer" containerID="f123ac3e083c408b31f992d541505e785759db4952252c657c95422bd0862913" Nov 03 00:05:23.540990 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.540948 5076 scope.go:122] "RemoveContainer" containerID="d99611eee0218a1928e4a81cd7dc524b3fd2340685a3fefc42c6c1cd09bfc6bd" Nov 03 00:05:23.550216 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.550195 5076 scope.go:122] "RemoveContainer" containerID="f123ac3e083c408b31f992d541505e785759db4952252c657c95422bd0862913" Nov 03 00:05:23.550483 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:05:23.550464 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f123ac3e083c408b31f992d541505e785759db4952252c657c95422bd0862913\": container with ID starting with f123ac3e083c408b31f992d541505e785759db4952252c657c95422bd0862913 not found: ID does not exist" containerID="f123ac3e083c408b31f992d541505e785759db4952252c657c95422bd0862913" Nov 03 00:05:23.550535 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.550493 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f123ac3e083c408b31f992d541505e785759db4952252c657c95422bd0862913"} err="failed to get container status \"f123ac3e083c408b31f992d541505e785759db4952252c657c95422bd0862913\": rpc error: code = NotFound desc = could not find container \"f123ac3e083c408b31f992d541505e785759db4952252c657c95422bd0862913\": container with ID starting with f123ac3e083c408b31f992d541505e785759db4952252c657c95422bd0862913 not found: ID does not exist" Nov 03 00:05:23.550535 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.550513 5076 scope.go:122] "RemoveContainer" containerID="d99611eee0218a1928e4a81cd7dc524b3fd2340685a3fefc42c6c1cd09bfc6bd" Nov 03 00:05:23.550747 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:05:23.550728 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d99611eee0218a1928e4a81cd7dc524b3fd2340685a3fefc42c6c1cd09bfc6bd\": container with ID starting with d99611eee0218a1928e4a81cd7dc524b3fd2340685a3fefc42c6c1cd09bfc6bd not found: ID does not exist" containerID="d99611eee0218a1928e4a81cd7dc524b3fd2340685a3fefc42c6c1cd09bfc6bd" Nov 03 00:05:23.550808 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.550757 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d99611eee0218a1928e4a81cd7dc524b3fd2340685a3fefc42c6c1cd09bfc6bd"} err="failed to get container status \"d99611eee0218a1928e4a81cd7dc524b3fd2340685a3fefc42c6c1cd09bfc6bd\": rpc error: code = NotFound desc = could not find container \"d99611eee0218a1928e4a81cd7dc524b3fd2340685a3fefc42c6c1cd09bfc6bd\": container with ID starting with d99611eee0218a1928e4a81cd7dc524b3fd2340685a3fefc42c6c1cd09bfc6bd not found: ID does not exist" Nov 03 00:05:23.686429 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.686270 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb"] Nov 03 00:05:23.724009 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:23.723976 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e1be81-openshift-default-79d9b7bd6c-cqhsb"] Nov 03 00:05:25.151334 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:05:25.151286 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df94f65b-8ef5-4e2c-a749-93a97b9c68e0" path="/var/lib/kubelet/pods/df94f65b-8ef5-4e2c-a749-93a97b9c68e0/volumes" Nov 03 00:05:39.887209 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:05:39.887169 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:06:09.882971 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:06:09.882927 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:06:33.347312 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:06:33.347273 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx"] Nov 03 00:06:33.347842 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:06:33.347476 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx" podUID="88a21350-c1be-4ac7-9f65-87fc3c18c136" containerName="maas-api" containerID="cri-o://c5b6d556dcf9ee27153d5eafdda7d25769a04e8e80c6883f042a0fbbffc97a07" gracePeriod=30 Nov 03 00:06:33.655821 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:06:33.655794 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx" Nov 03 00:06:33.819871 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:06:33.819832 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88a21350-c1be-4ac7-9f65-87fc3c18c136-kube-api-access-dggwn\" (UniqueName: \"kubernetes.io/projected/88a21350-c1be-4ac7-9f65-87fc3c18c136-kube-api-access-dggwn\") pod \"88a21350-c1be-4ac7-9f65-87fc3c18c136\" (UID: \"88a21350-c1be-4ac7-9f65-87fc3c18c136\") " Nov 03 00:06:33.820084 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:06:33.819989 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88a21350-c1be-4ac7-9f65-87fc3c18c136-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88a21350-c1be-4ac7-9f65-87fc3c18c136-maas-api-tls\") pod \"88a21350-c1be-4ac7-9f65-87fc3c18c136\" (UID: \"88a21350-c1be-4ac7-9f65-87fc3c18c136\") " Nov 03 00:06:33.820167 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:06:33.820149 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88a21350-c1be-4ac7-9f65-87fc3c18c136-metrics-certs\" (UniqueName: \"kubernetes.io/secret/88a21350-c1be-4ac7-9f65-87fc3c18c136-metrics-certs\") pod \"88a21350-c1be-4ac7-9f65-87fc3c18c136\" (UID: \"88a21350-c1be-4ac7-9f65-87fc3c18c136\") " Nov 03 00:06:33.820220 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:06:33.820184 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88a21350-c1be-4ac7-9f65-87fc3c18c136-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88a21350-c1be-4ac7-9f65-87fc3c18c136-openshift-service-ca\") pod \"88a21350-c1be-4ac7-9f65-87fc3c18c136\" (UID: \"88a21350-c1be-4ac7-9f65-87fc3c18c136\") " Nov 03 00:06:33.820658 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:06:33.820631 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88a21350-c1be-4ac7-9f65-87fc3c18c136-openshift-service-ca" pod "88a21350-c1be-4ac7-9f65-87fc3c18c136" (UID: "88a21350-c1be-4ac7-9f65-87fc3c18c136"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:33.821169 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:06:33.821147 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88a21350-c1be-4ac7-9f65-87fc3c18c136-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:06:33.823673 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:06:33.823652 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88a21350-c1be-4ac7-9f65-87fc3c18c136-kube-api-access-dggwn" pod "88a21350-c1be-4ac7-9f65-87fc3c18c136" (UID: "88a21350-c1be-4ac7-9f65-87fc3c18c136"). InnerVolumeSpecName "kube-api-access-dggwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:33.823827 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:06:33.823650 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88a21350-c1be-4ac7-9f65-87fc3c18c136-maas-api-tls" pod "88a21350-c1be-4ac7-9f65-87fc3c18c136" (UID: "88a21350-c1be-4ac7-9f65-87fc3c18c136"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:33.825240 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:06:33.825215 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88a21350-c1be-4ac7-9f65-87fc3c18c136-metrics-certs" pod "88a21350-c1be-4ac7-9f65-87fc3c18c136" (UID: "88a21350-c1be-4ac7-9f65-87fc3c18c136"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:33.928965 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:06:33.928199 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dggwn\" (UniqueName: \"kubernetes.io/projected/88a21350-c1be-4ac7-9f65-87fc3c18c136-kube-api-access-dggwn\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:06:33.928965 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:06:33.928242 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88a21350-c1be-4ac7-9f65-87fc3c18c136-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:06:33.928965 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:06:33.928256 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88a21350-c1be-4ac7-9f65-87fc3c18c136-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:06:34.019776 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:06:34.019740 5076 generic.go:356] "Generic (PLEG): container finished" podID="88a21350-c1be-4ac7-9f65-87fc3c18c136" containerID="c5b6d556dcf9ee27153d5eafdda7d25769a04e8e80c6883f042a0fbbffc97a07" exitCode=0 Nov 03 00:06:34.020028 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:06:34.019784 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx" Nov 03 00:06:34.020028 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:06:34.019789 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx" event={"ID":"88a21350-c1be-4ac7-9f65-87fc3c18c136","Type":"ContainerDied","Data":"c5b6d556dcf9ee27153d5eafdda7d25769a04e8e80c6883f042a0fbbffc97a07"} Nov 03 00:06:34.020028 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:06:34.019944 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx" event={"ID":"88a21350-c1be-4ac7-9f65-87fc3c18c136","Type":"ContainerDied","Data":"16f6b5452352cccf85967bfc9a00602a4a4ac19779e38b99966b9d9b73d50b82"} Nov 03 00:06:34.020028 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:06:34.019984 5076 scope.go:122] "RemoveContainer" containerID="c5b6d556dcf9ee27153d5eafdda7d25769a04e8e80c6883f042a0fbbffc97a07" Nov 03 00:06:34.031793 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:06:34.031773 5076 scope.go:122] "RemoveContainer" containerID="c5b6d556dcf9ee27153d5eafdda7d25769a04e8e80c6883f042a0fbbffc97a07" Nov 03 00:06:34.032127 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:06:34.032105 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5b6d556dcf9ee27153d5eafdda7d25769a04e8e80c6883f042a0fbbffc97a07\": container with ID starting with c5b6d556dcf9ee27153d5eafdda7d25769a04e8e80c6883f042a0fbbffc97a07 not found: ID does not exist" containerID="c5b6d556dcf9ee27153d5eafdda7d25769a04e8e80c6883f042a0fbbffc97a07" Nov 03 00:06:34.032234 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:06:34.032135 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5b6d556dcf9ee27153d5eafdda7d25769a04e8e80c6883f042a0fbbffc97a07"} err="failed to get container status \"c5b6d556dcf9ee27153d5eafdda7d25769a04e8e80c6883f042a0fbbffc97a07\": rpc error: code = NotFound desc = could not find container \"c5b6d556dcf9ee27153d5eafdda7d25769a04e8e80c6883f042a0fbbffc97a07\": container with ID starting with c5b6d556dcf9ee27153d5eafdda7d25769a04e8e80c6883f042a0fbbffc97a07 not found: ID does not exist" Nov 03 00:06:34.176608 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:06:34.176564 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx"] Nov 03 00:06:34.182166 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:06:34.182102 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-68d89f-56bb7d6574-6bbsx"] Nov 03 00:06:35.148713 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:06:35.148679 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88a21350-c1be-4ac7-9f65-87fc3c18c136" path="/var/lib/kubelet/pods/88a21350-c1be-4ac7-9f65-87fc3c18c136/volumes" Nov 03 00:06:39.887988 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:06:39.887950 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:07:09.883979 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:07:09.883868 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:07:29.154619 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:29.154583 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf"] Nov 03 00:07:29.155227 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:29.154802 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" podUID="0af00e3b-e17f-4ebe-b17b-740f4f185417" containerName="istio-proxy" containerID="cri-o://ed94b0c0bb72adc11dd8bea8bf8134a04b4bd003e0db3f9cbe4f01d0941c6a60" gracePeriod=30 Nov 03 00:07:32.104368 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:32.104340 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 03 00:07:32.195312 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:32.195284 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 03 00:07:34.418049 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:34.417983 5076 generic.go:356] "Generic (PLEG): container finished" podID="0af00e3b-e17f-4ebe-b17b-740f4f185417" containerID="ed94b0c0bb72adc11dd8bea8bf8134a04b4bd003e0db3f9cbe4f01d0941c6a60" exitCode=0 Nov 03 00:07:34.418049 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:34.418034 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" event={"ID":"0af00e3b-e17f-4ebe-b17b-740f4f185417","Type":"ContainerDied","Data":"ed94b0c0bb72adc11dd8bea8bf8134a04b4bd003e0db3f9cbe4f01d0941c6a60"} Nov 03 00:07:34.418440 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:34.418075 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" event={"ID":"0af00e3b-e17f-4ebe-b17b-740f4f185417","Type":"ContainerDied","Data":"70791d38b32a91adec123b6d02d30b320e259338b54aea00d26c6579fcaffe2a"} Nov 03 00:07:34.418440 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:34.418094 5076 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70791d38b32a91adec123b6d02d30b320e259338b54aea00d26c6579fcaffe2a" Nov 03 00:07:34.418440 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:34.418106 5076 scope.go:122] "RemoveContainer" containerID="5369931aad2f81b9b11e71153bacb8801036060ce3b812d0fee174ea2699da68" Nov 03 00:07:34.418440 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:34.418306 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" Nov 03 00:07:34.576946 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:34.576875 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0af00e3b-e17f-4ebe-b17b-740f4f185417-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0af00e3b-e17f-4ebe-b17b-740f4f185417-istiod-ca-cert\") pod \"0af00e3b-e17f-4ebe-b17b-740f4f185417\" (UID: \"0af00e3b-e17f-4ebe-b17b-740f4f185417\") " Nov 03 00:07:34.577160 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:34.577037 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0af00e3b-e17f-4ebe-b17b-740f4f185417-kube-api-access-p9qdq\" (UniqueName: \"kubernetes.io/projected/0af00e3b-e17f-4ebe-b17b-740f4f185417-kube-api-access-p9qdq\") pod \"0af00e3b-e17f-4ebe-b17b-740f4f185417\" (UID: \"0af00e3b-e17f-4ebe-b17b-740f4f185417\") " Nov 03 00:07:34.577160 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:34.577116 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0af00e3b-e17f-4ebe-b17b-740f4f185417-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0af00e3b-e17f-4ebe-b17b-740f4f185417-istio-podinfo\") pod \"0af00e3b-e17f-4ebe-b17b-740f4f185417\" (UID: \"0af00e3b-e17f-4ebe-b17b-740f4f185417\") " Nov 03 00:07:34.577367 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:34.577322 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0af00e3b-e17f-4ebe-b17b-740f4f185417-istiod-ca-cert" pod "0af00e3b-e17f-4ebe-b17b-740f4f185417" (UID: "0af00e3b-e17f-4ebe-b17b-740f4f185417"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:07:34.578058 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:34.578027 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0af00e3b-e17f-4ebe-b17b-740f4f185417-istio-token\" (UniqueName: \"kubernetes.io/projected/0af00e3b-e17f-4ebe-b17b-740f4f185417-istio-token\") pod \"0af00e3b-e17f-4ebe-b17b-740f4f185417\" (UID: \"0af00e3b-e17f-4ebe-b17b-740f4f185417\") " Nov 03 00:07:34.578227 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:34.578095 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0af00e3b-e17f-4ebe-b17b-740f4f185417-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0af00e3b-e17f-4ebe-b17b-740f4f185417-credential-socket\") pod \"0af00e3b-e17f-4ebe-b17b-740f4f185417\" (UID: \"0af00e3b-e17f-4ebe-b17b-740f4f185417\") " Nov 03 00:07:34.578227 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:34.578152 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0af00e3b-e17f-4ebe-b17b-740f4f185417-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0af00e3b-e17f-4ebe-b17b-740f4f185417-workload-socket\") pod \"0af00e3b-e17f-4ebe-b17b-740f4f185417\" (UID: \"0af00e3b-e17f-4ebe-b17b-740f4f185417\") " Nov 03 00:07:34.578428 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:34.578231 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0af00e3b-e17f-4ebe-b17b-740f4f185417-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0af00e3b-e17f-4ebe-b17b-740f4f185417-istio-data\") pod \"0af00e3b-e17f-4ebe-b17b-740f4f185417\" (UID: \"0af00e3b-e17f-4ebe-b17b-740f4f185417\") " Nov 03 00:07:34.578428 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:34.578309 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0af00e3b-e17f-4ebe-b17b-740f4f185417-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0af00e3b-e17f-4ebe-b17b-740f4f185417-istio-envoy\") pod \"0af00e3b-e17f-4ebe-b17b-740f4f185417\" (UID: \"0af00e3b-e17f-4ebe-b17b-740f4f185417\") " Nov 03 00:07:34.578428 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:34.578342 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0af00e3b-e17f-4ebe-b17b-740f4f185417-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0af00e3b-e17f-4ebe-b17b-740f4f185417-workload-certs\") pod \"0af00e3b-e17f-4ebe-b17b-740f4f185417\" (UID: \"0af00e3b-e17f-4ebe-b17b-740f4f185417\") " Nov 03 00:07:34.580361 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:34.578623 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af00e3b-e17f-4ebe-b17b-740f4f185417-workload-socket" pod "0af00e3b-e17f-4ebe-b17b-740f4f185417" (UID: "0af00e3b-e17f-4ebe-b17b-740f4f185417"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:07:34.580361 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:34.578674 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af00e3b-e17f-4ebe-b17b-740f4f185417-istio-data" pod "0af00e3b-e17f-4ebe-b17b-740f4f185417" (UID: "0af00e3b-e17f-4ebe-b17b-740f4f185417"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:07:34.580361 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:34.578856 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af00e3b-e17f-4ebe-b17b-740f4f185417-workload-certs" pod "0af00e3b-e17f-4ebe-b17b-740f4f185417" (UID: "0af00e3b-e17f-4ebe-b17b-740f4f185417"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:07:34.580361 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:34.579030 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0af00e3b-e17f-4ebe-b17b-740f4f185417-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:07:34.580361 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:34.579054 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0af00e3b-e17f-4ebe-b17b-740f4f185417-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:07:34.580361 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:34.579070 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0af00e3b-e17f-4ebe-b17b-740f4f185417-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:07:34.580361 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:34.579089 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0af00e3b-e17f-4ebe-b17b-740f4f185417-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:07:34.580361 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:34.579148 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af00e3b-e17f-4ebe-b17b-740f4f185417-credential-socket" pod "0af00e3b-e17f-4ebe-b17b-740f4f185417" (UID: "0af00e3b-e17f-4ebe-b17b-740f4f185417"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:07:34.580869 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:34.580799 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0af00e3b-e17f-4ebe-b17b-740f4f185417-istio-podinfo" pod "0af00e3b-e17f-4ebe-b17b-740f4f185417" (UID: "0af00e3b-e17f-4ebe-b17b-740f4f185417"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:07:34.580960 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:34.580911 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0af00e3b-e17f-4ebe-b17b-740f4f185417-kube-api-access-p9qdq" pod "0af00e3b-e17f-4ebe-b17b-740f4f185417" (UID: "0af00e3b-e17f-4ebe-b17b-740f4f185417"). InnerVolumeSpecName "kube-api-access-p9qdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:07:34.581082 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:34.581061 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0af00e3b-e17f-4ebe-b17b-740f4f185417-istio-token" pod "0af00e3b-e17f-4ebe-b17b-740f4f185417" (UID: "0af00e3b-e17f-4ebe-b17b-740f4f185417"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:07:34.582148 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:34.582120 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af00e3b-e17f-4ebe-b17b-740f4f185417-istio-envoy" pod "0af00e3b-e17f-4ebe-b17b-740f4f185417" (UID: "0af00e3b-e17f-4ebe-b17b-740f4f185417"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:07:34.680120 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:34.680038 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0af00e3b-e17f-4ebe-b17b-740f4f185417-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:07:34.680120 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:34.680069 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9qdq\" (UniqueName: \"kubernetes.io/projected/0af00e3b-e17f-4ebe-b17b-740f4f185417-kube-api-access-p9qdq\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:07:34.680120 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:34.680081 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0af00e3b-e17f-4ebe-b17b-740f4f185417-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:07:34.680120 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:34.680090 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0af00e3b-e17f-4ebe-b17b-740f4f185417-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:07:34.680120 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:34.680098 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0af00e3b-e17f-4ebe-b17b-740f4f185417-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:07:35.426023 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:35.425917 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf" Nov 03 00:07:35.693285 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:35.693247 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf"] Nov 03 00:07:35.696736 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:35.696711 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-886rf"] Nov 03 00:07:37.148882 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:37.148841 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0af00e3b-e17f-4ebe-b17b-740f4f185417" path="/var/lib/kubelet/pods/0af00e3b-e17f-4ebe-b17b-740f4f185417/volumes" Nov 03 00:07:39.886359 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:07:39.886315 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:07:40.975078 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:40.975041 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87"] Nov 03 00:07:40.975585 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:40.975305 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87" podUID="7917494a-587f-44da-b2f8-982a25c58526" containerName="maas-api" containerID="cri-o://df44a0ecd9ad6a87358c41dad096394f4b2f24cf71a8c7d387686bc87ace3e51" gracePeriod=30 Nov 03 00:07:41.299044 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:41.299016 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87" Nov 03 00:07:41.466177 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:41.466141 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7917494a-587f-44da-b2f8-982a25c58526-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7917494a-587f-44da-b2f8-982a25c58526-maas-api-tls\") pod \"7917494a-587f-44da-b2f8-982a25c58526\" (UID: \"7917494a-587f-44da-b2f8-982a25c58526\") " Nov 03 00:07:41.466396 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:41.466247 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7917494a-587f-44da-b2f8-982a25c58526-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7917494a-587f-44da-b2f8-982a25c58526-openshift-service-ca\") pod \"7917494a-587f-44da-b2f8-982a25c58526\" (UID: \"7917494a-587f-44da-b2f8-982a25c58526\") " Nov 03 00:07:41.466396 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:41.466373 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7917494a-587f-44da-b2f8-982a25c58526-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7917494a-587f-44da-b2f8-982a25c58526-metrics-certs\") pod \"7917494a-587f-44da-b2f8-982a25c58526\" (UID: \"7917494a-587f-44da-b2f8-982a25c58526\") " Nov 03 00:07:41.466544 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:41.466409 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7917494a-587f-44da-b2f8-982a25c58526-kube-api-access-w6sws\" (UniqueName: \"kubernetes.io/projected/7917494a-587f-44da-b2f8-982a25c58526-kube-api-access-w6sws\") pod \"7917494a-587f-44da-b2f8-982a25c58526\" (UID: \"7917494a-587f-44da-b2f8-982a25c58526\") " Nov 03 00:07:41.466720 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:41.466683 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7917494a-587f-44da-b2f8-982a25c58526-openshift-service-ca" pod "7917494a-587f-44da-b2f8-982a25c58526" (UID: "7917494a-587f-44da-b2f8-982a25c58526"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:07:41.466865 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:41.466837 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7917494a-587f-44da-b2f8-982a25c58526-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:07:41.468167 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:41.468134 5076 generic.go:356] "Generic (PLEG): container finished" podID="7917494a-587f-44da-b2f8-982a25c58526" containerID="df44a0ecd9ad6a87358c41dad096394f4b2f24cf71a8c7d387686bc87ace3e51" exitCode=0 Nov 03 00:07:41.468290 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:41.468197 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87" Nov 03 00:07:41.468290 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:41.468219 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87" event={"ID":"7917494a-587f-44da-b2f8-982a25c58526","Type":"ContainerDied","Data":"df44a0ecd9ad6a87358c41dad096394f4b2f24cf71a8c7d387686bc87ace3e51"} Nov 03 00:07:41.468402 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:41.468291 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87" event={"ID":"7917494a-587f-44da-b2f8-982a25c58526","Type":"ContainerDied","Data":"5dfd1e8cf94044286d8924e1b579825eb696a8ac1f82a114a3e7fade9cd06cb9"} Nov 03 00:07:41.468402 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:41.468325 5076 scope.go:122] "RemoveContainer" containerID="df44a0ecd9ad6a87358c41dad096394f4b2f24cf71a8c7d387686bc87ace3e51" Nov 03 00:07:41.470252 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:41.470224 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7917494a-587f-44da-b2f8-982a25c58526-maas-api-tls" pod "7917494a-587f-44da-b2f8-982a25c58526" (UID: "7917494a-587f-44da-b2f8-982a25c58526"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:07:41.470252 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:41.470249 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7917494a-587f-44da-b2f8-982a25c58526-metrics-certs" pod "7917494a-587f-44da-b2f8-982a25c58526" (UID: "7917494a-587f-44da-b2f8-982a25c58526"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:07:41.472331 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:41.472304 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7917494a-587f-44da-b2f8-982a25c58526-kube-api-access-w6sws" pod "7917494a-587f-44da-b2f8-982a25c58526" (UID: "7917494a-587f-44da-b2f8-982a25c58526"). InnerVolumeSpecName "kube-api-access-w6sws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:07:41.497647 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:41.497621 5076 scope.go:122] "RemoveContainer" containerID="df44a0ecd9ad6a87358c41dad096394f4b2f24cf71a8c7d387686bc87ace3e51" Nov 03 00:07:41.498007 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:07:41.497985 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df44a0ecd9ad6a87358c41dad096394f4b2f24cf71a8c7d387686bc87ace3e51\": container with ID starting with df44a0ecd9ad6a87358c41dad096394f4b2f24cf71a8c7d387686bc87ace3e51 not found: ID does not exist" containerID="df44a0ecd9ad6a87358c41dad096394f4b2f24cf71a8c7d387686bc87ace3e51" Nov 03 00:07:41.498078 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:41.498018 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df44a0ecd9ad6a87358c41dad096394f4b2f24cf71a8c7d387686bc87ace3e51"} err="failed to get container status \"df44a0ecd9ad6a87358c41dad096394f4b2f24cf71a8c7d387686bc87ace3e51\": rpc error: code = NotFound desc = could not find container \"df44a0ecd9ad6a87358c41dad096394f4b2f24cf71a8c7d387686bc87ace3e51\": container with ID starting with df44a0ecd9ad6a87358c41dad096394f4b2f24cf71a8c7d387686bc87ace3e51 not found: ID does not exist" Nov 03 00:07:41.568669 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:41.568575 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7917494a-587f-44da-b2f8-982a25c58526-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:07:41.569053 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:41.569030 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6sws\" (UniqueName: \"kubernetes.io/projected/7917494a-587f-44da-b2f8-982a25c58526-kube-api-access-w6sws\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:07:41.569590 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:41.569505 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7917494a-587f-44da-b2f8-982a25c58526-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:07:41.908972 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:41.899596 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87"] Nov 03 00:07:41.938913 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:41.926787 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-666a38-74c7fd8c8d-5qb87"] Nov 03 00:07:43.153375 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:43.153339 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7917494a-587f-44da-b2f8-982a25c58526" path="/var/lib/kubelet/pods/7917494a-587f-44da-b2f8-982a25c58526/volumes" Nov 03 00:07:51.915508 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:51.915470 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh"] Nov 03 00:07:51.916042 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:51.915717 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh" podUID="ff048ceb-36d0-4d93-b323-aa8aa6e1c16a" containerName="maas-api" containerID="cri-o://0128d229f5e9a49ef71d9c6dc5e7084513a22d045012bbb52ec2a6c4fe3385bd" gracePeriod=30 Nov 03 00:07:52.229479 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:52.229448 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh" Nov 03 00:07:52.316927 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:52.303455 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff048ceb-36d0-4d93-b323-aa8aa6e1c16a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff048ceb-36d0-4d93-b323-aa8aa6e1c16a-maas-api-tls\") pod \"ff048ceb-36d0-4d93-b323-aa8aa6e1c16a\" (UID: \"ff048ceb-36d0-4d93-b323-aa8aa6e1c16a\") " Nov 03 00:07:52.316927 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:52.303548 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff048ceb-36d0-4d93-b323-aa8aa6e1c16a-kube-api-access-lgdbm\" (UniqueName: \"kubernetes.io/projected/ff048ceb-36d0-4d93-b323-aa8aa6e1c16a-kube-api-access-lgdbm\") pod \"ff048ceb-36d0-4d93-b323-aa8aa6e1c16a\" (UID: \"ff048ceb-36d0-4d93-b323-aa8aa6e1c16a\") " Nov 03 00:07:52.316927 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:52.303656 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff048ceb-36d0-4d93-b323-aa8aa6e1c16a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff048ceb-36d0-4d93-b323-aa8aa6e1c16a-metrics-certs\") pod \"ff048ceb-36d0-4d93-b323-aa8aa6e1c16a\" (UID: \"ff048ceb-36d0-4d93-b323-aa8aa6e1c16a\") " Nov 03 00:07:52.316927 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:52.303713 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff048ceb-36d0-4d93-b323-aa8aa6e1c16a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff048ceb-36d0-4d93-b323-aa8aa6e1c16a-openshift-service-ca\") pod \"ff048ceb-36d0-4d93-b323-aa8aa6e1c16a\" (UID: \"ff048ceb-36d0-4d93-b323-aa8aa6e1c16a\") " Nov 03 00:07:52.322918 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:52.319620 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff048ceb-36d0-4d93-b323-aa8aa6e1c16a-openshift-service-ca" pod "ff048ceb-36d0-4d93-b323-aa8aa6e1c16a" (UID: "ff048ceb-36d0-4d93-b323-aa8aa6e1c16a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:07:52.329139 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:52.329063 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff048ceb-36d0-4d93-b323-aa8aa6e1c16a-maas-api-tls" pod "ff048ceb-36d0-4d93-b323-aa8aa6e1c16a" (UID: "ff048ceb-36d0-4d93-b323-aa8aa6e1c16a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:07:52.356755 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:52.346700 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff048ceb-36d0-4d93-b323-aa8aa6e1c16a-kube-api-access-lgdbm" pod "ff048ceb-36d0-4d93-b323-aa8aa6e1c16a" (UID: "ff048ceb-36d0-4d93-b323-aa8aa6e1c16a"). InnerVolumeSpecName "kube-api-access-lgdbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:07:52.358975 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:52.358767 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff048ceb-36d0-4d93-b323-aa8aa6e1c16a-metrics-certs" pod "ff048ceb-36d0-4d93-b323-aa8aa6e1c16a" (UID: "ff048ceb-36d0-4d93-b323-aa8aa6e1c16a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:07:52.408568 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:52.408535 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff048ceb-36d0-4d93-b323-aa8aa6e1c16a-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:07:52.408568 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:52.408565 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgdbm\" (UniqueName: \"kubernetes.io/projected/ff048ceb-36d0-4d93-b323-aa8aa6e1c16a-kube-api-access-lgdbm\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:07:52.408772 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:52.408579 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff048ceb-36d0-4d93-b323-aa8aa6e1c16a-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:07:52.408772 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:52.408593 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff048ceb-36d0-4d93-b323-aa8aa6e1c16a-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:07:52.544419 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:52.544327 5076 generic.go:356] "Generic (PLEG): container finished" podID="ff048ceb-36d0-4d93-b323-aa8aa6e1c16a" containerID="0128d229f5e9a49ef71d9c6dc5e7084513a22d045012bbb52ec2a6c4fe3385bd" exitCode=0 Nov 03 00:07:52.544419 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:52.544375 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh" Nov 03 00:07:52.544618 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:52.544412 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh" event={"ID":"ff048ceb-36d0-4d93-b323-aa8aa6e1c16a","Type":"ContainerDied","Data":"0128d229f5e9a49ef71d9c6dc5e7084513a22d045012bbb52ec2a6c4fe3385bd"} Nov 03 00:07:52.544618 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:52.544491 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh" event={"ID":"ff048ceb-36d0-4d93-b323-aa8aa6e1c16a","Type":"ContainerDied","Data":"afdb454ce08cc5670824a3aee3e70b6c4527542c9b09beed29d7f340f4386cc4"} Nov 03 00:07:52.544618 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:52.544512 5076 scope.go:122] "RemoveContainer" containerID="0128d229f5e9a49ef71d9c6dc5e7084513a22d045012bbb52ec2a6c4fe3385bd" Nov 03 00:07:52.557028 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:52.557000 5076 scope.go:122] "RemoveContainer" containerID="0128d229f5e9a49ef71d9c6dc5e7084513a22d045012bbb52ec2a6c4fe3385bd" Nov 03 00:07:52.557667 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:07:52.557645 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0128d229f5e9a49ef71d9c6dc5e7084513a22d045012bbb52ec2a6c4fe3385bd\": container with ID starting with 0128d229f5e9a49ef71d9c6dc5e7084513a22d045012bbb52ec2a6c4fe3385bd not found: ID does not exist" containerID="0128d229f5e9a49ef71d9c6dc5e7084513a22d045012bbb52ec2a6c4fe3385bd" Nov 03 00:07:52.557750 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:52.557676 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0128d229f5e9a49ef71d9c6dc5e7084513a22d045012bbb52ec2a6c4fe3385bd"} err="failed to get container status \"0128d229f5e9a49ef71d9c6dc5e7084513a22d045012bbb52ec2a6c4fe3385bd\": rpc error: code = NotFound desc = could not find container \"0128d229f5e9a49ef71d9c6dc5e7084513a22d045012bbb52ec2a6c4fe3385bd\": container with ID starting with 0128d229f5e9a49ef71d9c6dc5e7084513a22d045012bbb52ec2a6c4fe3385bd not found: ID does not exist" Nov 03 00:07:52.705358 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:52.705314 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh"] Nov 03 00:07:52.716505 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:52.716475 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6d61f9-5bb74df854-649kh"] Nov 03 00:07:53.163932 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:07:53.163283 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff048ceb-36d0-4d93-b323-aa8aa6e1c16a" path="/var/lib/kubelet/pods/ff048ceb-36d0-4d93-b323-aa8aa6e1c16a/volumes" Nov 03 00:08:09.410694 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.410658 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-fe63ac/rate-test-model-fe63ac-kserve-54cb8c857c-lntw7"] Nov 03 00:08:09.411236 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.411220 5076 cpu_manager.go:414] "containerMap: removing container" podUID="88a21350-c1be-4ac7-9f65-87fc3c18c136" containerName="maas-api" Nov 03 00:08:09.411286 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.411238 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88a21350-c1be-4ac7-9f65-87fc3c18c136" containerName="maas-api" Nov 03 00:08:09.411286 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.411250 5076 cpu_manager.go:414] "containerMap: removing container" podUID="df94f65b-8ef5-4e2c-a749-93a97b9c68e0" containerName="istio-proxy" Nov 03 00:08:09.411286 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.411255 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df94f65b-8ef5-4e2c-a749-93a97b9c68e0" containerName="istio-proxy" Nov 03 00:08:09.411286 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.411267 5076 cpu_manager.go:414] "containerMap: removing container" podUID="df94f65b-8ef5-4e2c-a749-93a97b9c68e0" containerName="istio-proxy" Nov 03 00:08:09.411286 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.411274 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df94f65b-8ef5-4e2c-a749-93a97b9c68e0" containerName="istio-proxy" Nov 03 00:08:09.411286 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.411280 5076 cpu_manager.go:414] "containerMap: removing container" podUID="bae87c4b-16e4-4fb3-a5db-cc529edebd50" containerName="istio-proxy" Nov 03 00:08:09.411286 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.411285 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bae87c4b-16e4-4fb3-a5db-cc529edebd50" containerName="istio-proxy" Nov 03 00:08:09.411505 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.411297 5076 cpu_manager.go:414] "containerMap: removing container" podUID="bae87c4b-16e4-4fb3-a5db-cc529edebd50" containerName="istio-proxy" Nov 03 00:08:09.411505 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.411302 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bae87c4b-16e4-4fb3-a5db-cc529edebd50" containerName="istio-proxy" Nov 03 00:08:09.411505 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.411318 5076 cpu_manager.go:414] "containerMap: removing container" podUID="22d4e172-af11-4e35-9363-c4959295db10" containerName="istio-proxy" Nov 03 00:08:09.411505 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.411323 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22d4e172-af11-4e35-9363-c4959295db10" containerName="istio-proxy" Nov 03 00:08:09.411505 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.411333 5076 cpu_manager.go:414] "containerMap: removing container" podUID="0af00e3b-e17f-4ebe-b17b-740f4f185417" containerName="istio-proxy" Nov 03 00:08:09.411505 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.411338 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0af00e3b-e17f-4ebe-b17b-740f4f185417" containerName="istio-proxy" Nov 03 00:08:09.411505 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.411346 5076 cpu_manager.go:414] "containerMap: removing container" podUID="7917494a-587f-44da-b2f8-982a25c58526" containerName="maas-api" Nov 03 00:08:09.411505 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.411350 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7917494a-587f-44da-b2f8-982a25c58526" containerName="maas-api" Nov 03 00:08:09.411505 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.411359 5076 cpu_manager.go:414] "containerMap: removing container" podUID="3533e1bf-11a4-4c94-a310-6edeaa98a834" containerName="maas-api" Nov 03 00:08:09.411505 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.411363 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3533e1bf-11a4-4c94-a310-6edeaa98a834" containerName="maas-api" Nov 03 00:08:09.411505 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.411375 5076 cpu_manager.go:414] "containerMap: removing container" podUID="ff048ceb-36d0-4d93-b323-aa8aa6e1c16a" containerName="maas-api" Nov 03 00:08:09.411505 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.411379 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff048ceb-36d0-4d93-b323-aa8aa6e1c16a" containerName="maas-api" Nov 03 00:08:09.411505 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.411391 5076 cpu_manager.go:414] "containerMap: removing container" podUID="9a09ac6d-1d69-4d01-8863-5e693b5533df" containerName="maas-api" Nov 03 00:08:09.411505 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.411396 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a09ac6d-1d69-4d01-8863-5e693b5533df" containerName="maas-api" Nov 03 00:08:09.411505 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.411471 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="bae87c4b-16e4-4fb3-a5db-cc529edebd50" containerName="istio-proxy" Nov 03 00:08:09.411505 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.411482 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="22d4e172-af11-4e35-9363-c4959295db10" containerName="istio-proxy" Nov 03 00:08:09.411505 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.411489 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="0af00e3b-e17f-4ebe-b17b-740f4f185417" containerName="istio-proxy" Nov 03 00:08:09.411505 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.411496 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="7917494a-587f-44da-b2f8-982a25c58526" containerName="maas-api" Nov 03 00:08:09.411505 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.411503 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="3533e1bf-11a4-4c94-a310-6edeaa98a834" containerName="maas-api" Nov 03 00:08:09.411505 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.411510 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="22d4e172-af11-4e35-9363-c4959295db10" containerName="istio-proxy" Nov 03 00:08:09.412147 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.411518 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff048ceb-36d0-4d93-b323-aa8aa6e1c16a" containerName="maas-api" Nov 03 00:08:09.412147 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.411533 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a09ac6d-1d69-4d01-8863-5e693b5533df" containerName="maas-api" Nov 03 00:08:09.412147 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.411538 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="0af00e3b-e17f-4ebe-b17b-740f4f185417" containerName="istio-proxy" Nov 03 00:08:09.412147 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.411546 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="88a21350-c1be-4ac7-9f65-87fc3c18c136" containerName="maas-api" Nov 03 00:08:09.412147 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.411551 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="df94f65b-8ef5-4e2c-a749-93a97b9c68e0" containerName="istio-proxy" Nov 03 00:08:09.412147 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.411557 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="df94f65b-8ef5-4e2c-a749-93a97b9c68e0" containerName="istio-proxy" Nov 03 00:08:09.416462 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.416442 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-fe63ac/rate-test-model-fe63ac-kserve-54cb8c857c-lntw7" Nov 03 00:08:09.474123 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.474089 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-fe63ac/rate-test-model-fe63ac-kserve-54cb8c857c-lntw7"] Nov 03 00:08:09.516962 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.516929 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-fe63ac\"/\"openshift-service-ca.crt\"" Nov 03 00:08:09.516962 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.516945 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-fe63ac\"/\"kube-root-ca.crt\"" Nov 03 00:08:09.517202 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.516988 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-fe63ac\"/\"default-dockercfg-tmqkp\"" Nov 03 00:08:09.517202 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.517075 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-fe63ac\"/\"rate-test-model-fe63ac-kserve-self-signed-certs\"" Nov 03 00:08:09.542550 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.542524 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af4882d8-765e-4167-9a4a-06a7d77f7437-model-cache\") pod \"rate-test-model-fe63ac-kserve-54cb8c857c-lntw7\" (UID: \"af4882d8-765e-4167-9a4a-06a7d77f7437\") " pod="ai-tenant-e2e-shared-a-w1-fe63ac/rate-test-model-fe63ac-kserve-54cb8c857c-lntw7" Nov 03 00:08:09.542688 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.542566 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2skw\" (UniqueName: \"kubernetes.io/projected/af4882d8-765e-4167-9a4a-06a7d77f7437-kube-api-access-h2skw\") pod \"rate-test-model-fe63ac-kserve-54cb8c857c-lntw7\" (UID: \"af4882d8-765e-4167-9a4a-06a7d77f7437\") " pod="ai-tenant-e2e-shared-a-w1-fe63ac/rate-test-model-fe63ac-kserve-54cb8c857c-lntw7" Nov 03 00:08:09.542777 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.542756 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af4882d8-765e-4167-9a4a-06a7d77f7437-tls-certs\") pod \"rate-test-model-fe63ac-kserve-54cb8c857c-lntw7\" (UID: \"af4882d8-765e-4167-9a4a-06a7d77f7437\") " pod="ai-tenant-e2e-shared-a-w1-fe63ac/rate-test-model-fe63ac-kserve-54cb8c857c-lntw7" Nov 03 00:08:09.542819 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.542789 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af4882d8-765e-4167-9a4a-06a7d77f7437-home\") pod \"rate-test-model-fe63ac-kserve-54cb8c857c-lntw7\" (UID: \"af4882d8-765e-4167-9a4a-06a7d77f7437\") " pod="ai-tenant-e2e-shared-a-w1-fe63ac/rate-test-model-fe63ac-kserve-54cb8c857c-lntw7" Nov 03 00:08:09.542918 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.542881 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af4882d8-765e-4167-9a4a-06a7d77f7437-dshm\") pod \"rate-test-model-fe63ac-kserve-54cb8c857c-lntw7\" (UID: \"af4882d8-765e-4167-9a4a-06a7d77f7437\") " pod="ai-tenant-e2e-shared-a-w1-fe63ac/rate-test-model-fe63ac-kserve-54cb8c857c-lntw7" Nov 03 00:08:09.542962 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.542942 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af4882d8-765e-4167-9a4a-06a7d77f7437-tmp-dir\") pod \"rate-test-model-fe63ac-kserve-54cb8c857c-lntw7\" (UID: \"af4882d8-765e-4167-9a4a-06a7d77f7437\") " pod="ai-tenant-e2e-shared-a-w1-fe63ac/rate-test-model-fe63ac-kserve-54cb8c857c-lntw7" Nov 03 00:08:09.644619 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.644579 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2skw\" (UniqueName: \"kubernetes.io/projected/af4882d8-765e-4167-9a4a-06a7d77f7437-kube-api-access-h2skw\") pod \"rate-test-model-fe63ac-kserve-54cb8c857c-lntw7\" (UID: \"af4882d8-765e-4167-9a4a-06a7d77f7437\") " pod="ai-tenant-e2e-shared-a-w1-fe63ac/rate-test-model-fe63ac-kserve-54cb8c857c-lntw7" Nov 03 00:08:09.644828 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.644771 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af4882d8-765e-4167-9a4a-06a7d77f7437-tls-certs\") pod \"rate-test-model-fe63ac-kserve-54cb8c857c-lntw7\" (UID: \"af4882d8-765e-4167-9a4a-06a7d77f7437\") " pod="ai-tenant-e2e-shared-a-w1-fe63ac/rate-test-model-fe63ac-kserve-54cb8c857c-lntw7" Nov 03 00:08:09.644828 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.644801 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af4882d8-765e-4167-9a4a-06a7d77f7437-home\") pod \"rate-test-model-fe63ac-kserve-54cb8c857c-lntw7\" (UID: \"af4882d8-765e-4167-9a4a-06a7d77f7437\") " pod="ai-tenant-e2e-shared-a-w1-fe63ac/rate-test-model-fe63ac-kserve-54cb8c857c-lntw7" Nov 03 00:08:09.645016 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.644833 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af4882d8-765e-4167-9a4a-06a7d77f7437-dshm\") pod \"rate-test-model-fe63ac-kserve-54cb8c857c-lntw7\" (UID: \"af4882d8-765e-4167-9a4a-06a7d77f7437\") " pod="ai-tenant-e2e-shared-a-w1-fe63ac/rate-test-model-fe63ac-kserve-54cb8c857c-lntw7" Nov 03 00:08:09.645016 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.644852 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af4882d8-765e-4167-9a4a-06a7d77f7437-tmp-dir\") pod \"rate-test-model-fe63ac-kserve-54cb8c857c-lntw7\" (UID: \"af4882d8-765e-4167-9a4a-06a7d77f7437\") " pod="ai-tenant-e2e-shared-a-w1-fe63ac/rate-test-model-fe63ac-kserve-54cb8c857c-lntw7" Nov 03 00:08:09.645129 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.645018 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af4882d8-765e-4167-9a4a-06a7d77f7437-model-cache\") pod \"rate-test-model-fe63ac-kserve-54cb8c857c-lntw7\" (UID: \"af4882d8-765e-4167-9a4a-06a7d77f7437\") " pod="ai-tenant-e2e-shared-a-w1-fe63ac/rate-test-model-fe63ac-kserve-54cb8c857c-lntw7" Nov 03 00:08:09.645312 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.645285 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af4882d8-765e-4167-9a4a-06a7d77f7437-tmp-dir\") pod \"rate-test-model-fe63ac-kserve-54cb8c857c-lntw7\" (UID: \"af4882d8-765e-4167-9a4a-06a7d77f7437\") " pod="ai-tenant-e2e-shared-a-w1-fe63ac/rate-test-model-fe63ac-kserve-54cb8c857c-lntw7" Nov 03 00:08:09.645407 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.645347 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af4882d8-765e-4167-9a4a-06a7d77f7437-home\") pod \"rate-test-model-fe63ac-kserve-54cb8c857c-lntw7\" (UID: \"af4882d8-765e-4167-9a4a-06a7d77f7437\") " pod="ai-tenant-e2e-shared-a-w1-fe63ac/rate-test-model-fe63ac-kserve-54cb8c857c-lntw7" Nov 03 00:08:09.645453 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.645425 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af4882d8-765e-4167-9a4a-06a7d77f7437-model-cache\") pod \"rate-test-model-fe63ac-kserve-54cb8c857c-lntw7\" (UID: \"af4882d8-765e-4167-9a4a-06a7d77f7437\") " pod="ai-tenant-e2e-shared-a-w1-fe63ac/rate-test-model-fe63ac-kserve-54cb8c857c-lntw7" Nov 03 00:08:09.647319 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.647289 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af4882d8-765e-4167-9a4a-06a7d77f7437-dshm\") pod \"rate-test-model-fe63ac-kserve-54cb8c857c-lntw7\" (UID: \"af4882d8-765e-4167-9a4a-06a7d77f7437\") " pod="ai-tenant-e2e-shared-a-w1-fe63ac/rate-test-model-fe63ac-kserve-54cb8c857c-lntw7" Nov 03 00:08:09.647608 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.647591 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af4882d8-765e-4167-9a4a-06a7d77f7437-tls-certs\") pod \"rate-test-model-fe63ac-kserve-54cb8c857c-lntw7\" (UID: \"af4882d8-765e-4167-9a4a-06a7d77f7437\") " pod="ai-tenant-e2e-shared-a-w1-fe63ac/rate-test-model-fe63ac-kserve-54cb8c857c-lntw7" Nov 03 00:08:09.698842 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.698809 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2skw\" (UniqueName: \"kubernetes.io/projected/af4882d8-765e-4167-9a4a-06a7d77f7437-kube-api-access-h2skw\") pod \"rate-test-model-fe63ac-kserve-54cb8c857c-lntw7\" (UID: \"af4882d8-765e-4167-9a4a-06a7d77f7437\") " pod="ai-tenant-e2e-shared-a-w1-fe63ac/rate-test-model-fe63ac-kserve-54cb8c857c-lntw7" Nov 03 00:08:09.729105 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:09.729079 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-fe63ac/rate-test-model-fe63ac-kserve-54cb8c857c-lntw7" Nov 03 00:08:09.931406 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:08:09.931374 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:08:10.096465 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:10.096437 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-fe63ac/rate-test-model-fe63ac-kserve-54cb8c857c-lntw7"] Nov 03 00:08:10.097518 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:08:10.097490 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf4882d8_765e_4167_9a4a_06a7d77f7437.slice/crio-8e027565521314fd95ca1ad461df5d6f5fe52edb7961be37c4d5fe9aa4fd94b3 WatchSource:0}: Error finding container 8e027565521314fd95ca1ad461df5d6f5fe52edb7961be37c4d5fe9aa4fd94b3: Status 404 returned error can't find the container with id 8e027565521314fd95ca1ad461df5d6f5fe52edb7961be37c4d5fe9aa4fd94b3 Nov 03 00:08:10.660297 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:10.660254 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-fe63ac/rate-test-model-fe63ac-kserve-54cb8c857c-lntw7" event={"ID":"af4882d8-765e-4167-9a4a-06a7d77f7437","Type":"ContainerStarted","Data":"f8a26f5d8a0a4d08087e8d647a86b5c069877f116d1f337927e7b839ab95a29c"} Nov 03 00:08:10.660778 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:10.660313 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-fe63ac/rate-test-model-fe63ac-kserve-54cb8c857c-lntw7" event={"ID":"af4882d8-765e-4167-9a4a-06a7d77f7437","Type":"ContainerStarted","Data":"8e027565521314fd95ca1ad461df5d6f5fe52edb7961be37c4d5fe9aa4fd94b3"} Nov 03 00:08:10.729246 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:10.729213 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-fe63ac/rate-test-model-fe63ac-kserve-54cb8c857c-lntw7" Nov 03 00:08:10.774269 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:10.774175 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-fe63ac/rate-test-model-fe63ac-kserve-54cb8c857c-lntw7" podStartSLOduration=1.774154594 podStartE2EDuration="1.774154594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:08:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:08:10.772830043 +0000 UTC m=+4539.741361828" watchObservedRunningTime="2026-11-03 00:08:10.774154594 +0000 UTC m=+4539.742686381" Nov 03 00:08:19.729990 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:19.729904 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-fe63ac/rate-test-model-fe63ac-kserve-54cb8c857c-lntw7" Nov 03 00:08:19.742664 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:19.742638 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-fe63ac/rate-test-model-fe63ac-kserve-54cb8c857c-lntw7" Nov 03 00:08:20.741302 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:20.741272 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-fe63ac/rate-test-model-fe63ac-kserve-54cb8c857c-lntw7" Nov 03 00:08:32.681770 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:08:32.681733 5076 scope.go:122] "RemoveContainer" containerID="ed94b0c0bb72adc11dd8bea8bf8134a04b4bd003e0db3f9cbe4f01d0941c6a60" Nov 03 00:08:39.885332 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:08:39.885294 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:09:09.883578 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:09:09.883538 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:09:39.884471 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:09:39.884432 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:10:09.883259 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:10:09.883177 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:10:11.707684 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:11.707647 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl"] Nov 03 00:10:11.708469 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:11.708444 5076 cpu_manager.go:414] "containerMap: removing container" podUID="22d4e172-af11-4e35-9363-c4959295db10" containerName="istio-proxy" Nov 03 00:10:11.708575 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:11.708470 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22d4e172-af11-4e35-9363-c4959295db10" containerName="istio-proxy" Nov 03 00:10:11.708575 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:11.708491 5076 cpu_manager.go:414] "containerMap: removing container" podUID="0af00e3b-e17f-4ebe-b17b-740f4f185417" containerName="istio-proxy" Nov 03 00:10:11.708575 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:11.708498 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0af00e3b-e17f-4ebe-b17b-740f4f185417" containerName="istio-proxy" Nov 03 00:10:11.708735 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:11.708662 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="bae87c4b-16e4-4fb3-a5db-cc529edebd50" containerName="istio-proxy" Nov 03 00:10:11.712172 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:11.712148 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" Nov 03 00:10:11.783721 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:11.783676 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-1f668443-openshift-default-dockercfg-kbnql\"" Nov 03 00:10:11.812937 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:11.812906 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl"] Nov 03 00:10:11.821003 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:11.820967 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjdb9\" (UniqueName: \"kubernetes.io/projected/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-kube-api-access-vjdb9\") pod \"e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl\" (UID: \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\") " pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" Nov 03 00:10:11.821309 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:11.821286 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-istio-token\") pod \"e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl\" (UID: \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\") " pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" Nov 03 00:10:11.821636 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:11.821616 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-credential-socket\") pod \"e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl\" (UID: \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\") " pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" Nov 03 00:10:11.821797 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:11.821781 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-istio-podinfo\") pod \"e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl\" (UID: \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\") " pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" Nov 03 00:10:11.821946 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:11.821927 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-workload-certs\") pod \"e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl\" (UID: \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\") " pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" Nov 03 00:10:11.822038 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:11.821971 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-istio-envoy\") pod \"e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl\" (UID: \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\") " pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" Nov 03 00:10:11.822038 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:11.822019 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-istio-data\") pod \"e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl\" (UID: \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\") " pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" Nov 03 00:10:11.822331 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:11.822064 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-istiod-ca-cert\") pod \"e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl\" (UID: \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\") " pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" Nov 03 00:10:11.822331 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:11.822094 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-workload-socket\") pod \"e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl\" (UID: \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\") " pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" Nov 03 00:10:11.923819 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:11.923785 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjdb9\" (UniqueName: \"kubernetes.io/projected/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-kube-api-access-vjdb9\") pod \"e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl\" (UID: \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\") " pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" Nov 03 00:10:11.924053 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:11.923868 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-istio-token\") pod \"e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl\" (UID: \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\") " pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" Nov 03 00:10:11.924053 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:11.923944 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-credential-socket\") pod \"e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl\" (UID: \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\") " pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" Nov 03 00:10:11.924053 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:11.924009 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-istio-podinfo\") pod \"e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl\" (UID: \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\") " pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" Nov 03 00:10:11.924272 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:11.924055 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-workload-certs\") pod \"e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl\" (UID: \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\") " pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" Nov 03 00:10:11.924272 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:11.924087 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-istio-envoy\") pod \"e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl\" (UID: \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\") " pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" Nov 03 00:10:11.924272 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:11.924131 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-istio-data\") pod \"e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl\" (UID: \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\") " pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" Nov 03 00:10:11.924272 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:11.924176 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-istiod-ca-cert\") pod \"e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl\" (UID: \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\") " pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" Nov 03 00:10:11.924272 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:11.924211 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-workload-socket\") pod \"e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl\" (UID: \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\") " pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" Nov 03 00:10:11.924557 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:11.924353 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-credential-socket\") pod \"e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl\" (UID: \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\") " pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" Nov 03 00:10:11.924557 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:11.924434 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-workload-certs\") pod \"e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl\" (UID: \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\") " pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" Nov 03 00:10:11.924708 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:11.924592 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-istio-data\") pod \"e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl\" (UID: \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\") " pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" Nov 03 00:10:11.924708 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:11.924679 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-workload-socket\") pod \"e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl\" (UID: \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\") " pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" Nov 03 00:10:11.924844 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:11.924821 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-istiod-ca-cert\") pod \"e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl\" (UID: \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\") " pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" Nov 03 00:10:11.926844 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:11.926819 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-istio-envoy\") pod \"e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl\" (UID: \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\") " pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" Nov 03 00:10:11.927188 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:11.927168 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-istio-podinfo\") pod \"e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl\" (UID: \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\") " pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" Nov 03 00:10:12.009750 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:12.009635 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjdb9\" (UniqueName: \"kubernetes.io/projected/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-kube-api-access-vjdb9\") pod \"e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl\" (UID: \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\") " pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" Nov 03 00:10:12.026926 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:12.025953 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-istio-token\") pod \"e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl\" (UID: \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\") " pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" Nov 03 00:10:12.326038 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:12.325944 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" Nov 03 00:10:12.747419 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:12.747391 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl"] Nov 03 00:10:12.748744 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:10:12.748710 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae791d6_2d7e_4a26_9904_0b7afb962ff8.slice/crio-449e1e7efbc10e351f6d3252b0df706be0c3c32812f40942c82eaf3b2b61d5cd WatchSource:0}: Error finding container 449e1e7efbc10e351f6d3252b0df706be0c3c32812f40942c82eaf3b2b61d5cd: Status 404 returned error can't find the container with id 449e1e7efbc10e351f6d3252b0df706be0c3c32812f40942c82eaf3b2b61d5cd Nov 03 00:10:12.751364 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:12.751330 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:10:12.751480 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:12.751410 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:10:12.751480 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:12.751454 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:10:13.599236 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:13.599195 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" event={"ID":"6ae791d6-2d7e-4a26-9904-0b7afb962ff8","Type":"ContainerStarted","Data":"2b52e5d661a97e1ea6a7706f43b26aea9efad61bd78431d3b56fad3b5ea399e7"} Nov 03 00:10:13.599453 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:13.599281 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" event={"ID":"6ae791d6-2d7e-4a26-9904-0b7afb962ff8","Type":"ContainerStarted","Data":"449e1e7efbc10e351f6d3252b0df706be0c3c32812f40942c82eaf3b2b61d5cd"} Nov 03 00:10:14.326382 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:14.326347 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" Nov 03 00:10:14.332372 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:14.332347 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" Nov 03 00:10:14.491583 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:14.491527 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" podStartSLOduration=3.491506917 podStartE2EDuration="3.491506917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:10:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:10:13.803803649 +0000 UTC m=+4662.772335433" watchObservedRunningTime="2026-11-03 00:10:14.491506917 +0000 UTC m=+4663.460038700" Nov 03 00:10:14.605421 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:14.605347 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" Nov 03 00:10:14.606407 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:10:14.606388 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" Nov 03 00:10:39.887761 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:10:39.887719 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:11:03.750381 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:03.750343 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-28f82c/sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s"] Nov 03 00:11:03.755485 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:03.755461 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-28f82c/sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s" Nov 03 00:11:03.786607 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:03.786575 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-model-cache\") pod \"sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s\" (UID: \"d3c9fa49-400c-4d00-bc52-cfc65b24d3d3\") " pod="ai-tenant-e2e-shared-a-w3-28f82c/sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s" Nov 03 00:11:03.786799 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:03.786640 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-dshm\") pod \"sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s\" (UID: \"d3c9fa49-400c-4d00-bc52-cfc65b24d3d3\") " pod="ai-tenant-e2e-shared-a-w3-28f82c/sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s" Nov 03 00:11:03.786799 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:03.786742 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-tmp-dir\") pod \"sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s\" (UID: \"d3c9fa49-400c-4d00-bc52-cfc65b24d3d3\") " pod="ai-tenant-e2e-shared-a-w3-28f82c/sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s" Nov 03 00:11:03.786799 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:03.786771 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htlt6\" (UniqueName: \"kubernetes.io/projected/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-kube-api-access-htlt6\") pod \"sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s\" (UID: \"d3c9fa49-400c-4d00-bc52-cfc65b24d3d3\") " pod="ai-tenant-e2e-shared-a-w3-28f82c/sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s" Nov 03 00:11:03.787031 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:03.786856 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-tls-certs\") pod \"sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s\" (UID: \"d3c9fa49-400c-4d00-bc52-cfc65b24d3d3\") " pod="ai-tenant-e2e-shared-a-w3-28f82c/sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s" Nov 03 00:11:03.787031 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:03.786886 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-home\") pod \"sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s\" (UID: \"d3c9fa49-400c-4d00-bc52-cfc65b24d3d3\") " pod="ai-tenant-e2e-shared-a-w3-28f82c/sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s" Nov 03 00:11:03.806664 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:03.806629 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-28f82c/sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s"] Nov 03 00:11:03.849738 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:03.849696 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-28f82c\"/\"kube-root-ca.crt\"" Nov 03 00:11:03.849909 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:03.849744 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-28f82c\"/\"sub-test-model-28f82c-kserve-self-signed-certs\"" Nov 03 00:11:03.849909 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:03.849834 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-28f82c\"/\"openshift-service-ca.crt\"" Nov 03 00:11:03.852294 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:03.852275 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-28f82c\"/\"default-dockercfg-2z7l9\"" Nov 03 00:11:03.887724 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:03.887687 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-dshm\") pod \"sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s\" (UID: \"d3c9fa49-400c-4d00-bc52-cfc65b24d3d3\") " pod="ai-tenant-e2e-shared-a-w3-28f82c/sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s" Nov 03 00:11:03.887852 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:03.887780 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-tmp-dir\") pod \"sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s\" (UID: \"d3c9fa49-400c-4d00-bc52-cfc65b24d3d3\") " pod="ai-tenant-e2e-shared-a-w3-28f82c/sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s" Nov 03 00:11:03.887852 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:03.887816 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htlt6\" (UniqueName: \"kubernetes.io/projected/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-kube-api-access-htlt6\") pod \"sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s\" (UID: \"d3c9fa49-400c-4d00-bc52-cfc65b24d3d3\") " pod="ai-tenant-e2e-shared-a-w3-28f82c/sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s" Nov 03 00:11:03.888013 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:03.887866 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-tls-certs\") pod \"sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s\" (UID: \"d3c9fa49-400c-4d00-bc52-cfc65b24d3d3\") " pod="ai-tenant-e2e-shared-a-w3-28f82c/sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s" Nov 03 00:11:03.888013 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:03.887938 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-home\") pod \"sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s\" (UID: \"d3c9fa49-400c-4d00-bc52-cfc65b24d3d3\") " pod="ai-tenant-e2e-shared-a-w3-28f82c/sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s" Nov 03 00:11:03.888127 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:03.888108 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-model-cache\") pod \"sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s\" (UID: \"d3c9fa49-400c-4d00-bc52-cfc65b24d3d3\") " pod="ai-tenant-e2e-shared-a-w3-28f82c/sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s" Nov 03 00:11:03.888263 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:03.888240 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-tmp-dir\") pod \"sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s\" (UID: \"d3c9fa49-400c-4d00-bc52-cfc65b24d3d3\") " pod="ai-tenant-e2e-shared-a-w3-28f82c/sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s" Nov 03 00:11:03.888373 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:03.888353 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-home\") pod \"sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s\" (UID: \"d3c9fa49-400c-4d00-bc52-cfc65b24d3d3\") " pod="ai-tenant-e2e-shared-a-w3-28f82c/sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s" Nov 03 00:11:03.888433 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:03.888405 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-model-cache\") pod \"sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s\" (UID: \"d3c9fa49-400c-4d00-bc52-cfc65b24d3d3\") " pod="ai-tenant-e2e-shared-a-w3-28f82c/sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s" Nov 03 00:11:03.890135 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:03.890112 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-dshm\") pod \"sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s\" (UID: \"d3c9fa49-400c-4d00-bc52-cfc65b24d3d3\") " pod="ai-tenant-e2e-shared-a-w3-28f82c/sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s" Nov 03 00:11:03.890736 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:03.890700 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-tls-certs\") pod \"sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s\" (UID: \"d3c9fa49-400c-4d00-bc52-cfc65b24d3d3\") " pod="ai-tenant-e2e-shared-a-w3-28f82c/sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s" Nov 03 00:11:03.947394 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:03.947359 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htlt6\" (UniqueName: \"kubernetes.io/projected/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-kube-api-access-htlt6\") pod \"sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s\" (UID: \"d3c9fa49-400c-4d00-bc52-cfc65b24d3d3\") " pod="ai-tenant-e2e-shared-a-w3-28f82c/sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s" Nov 03 00:11:04.068180 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:04.068094 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-28f82c/sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s" Nov 03 00:11:04.436861 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:04.436829 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-28f82c/sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s"] Nov 03 00:11:04.437780 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:11:04.437755 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3c9fa49_400c_4d00_bc52_cfc65b24d3d3.slice/crio-391b265896af73ed81195b31f3b719a71172094744861b79a28bd9bd68c347f5 WatchSource:0}: Error finding container 391b265896af73ed81195b31f3b719a71172094744861b79a28bd9bd68c347f5: Status 404 returned error can't find the container with id 391b265896af73ed81195b31f3b719a71172094744861b79a28bd9bd68c347f5 Nov 03 00:11:04.985399 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:04.985363 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-28f82c/sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s" event={"ID":"d3c9fa49-400c-4d00-bc52-cfc65b24d3d3","Type":"ContainerStarted","Data":"76b7fcb50a6d4959efcc22792a02621501ef2b3abf5f8ca0275851e920f04af7"} Nov 03 00:11:04.985797 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:04.985407 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-28f82c/sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s" event={"ID":"d3c9fa49-400c-4d00-bc52-cfc65b24d3d3","Type":"ContainerStarted","Data":"391b265896af73ed81195b31f3b719a71172094744861b79a28bd9bd68c347f5"} Nov 03 00:11:05.069076 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:05.069038 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-28f82c/sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s" Nov 03 00:11:05.094517 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:05.094468 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-28f82c/sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s" podStartSLOduration=2.094454695 podStartE2EDuration="2.094454695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:11:05.092457354 +0000 UTC m=+4714.060989141" watchObservedRunningTime="2026-11-03 00:11:05.094454695 +0000 UTC m=+4714.062986479" Nov 03 00:11:09.885860 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:11:09.885819 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:11:14.068505 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:14.068426 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-28f82c/sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s" Nov 03 00:11:14.081279 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:14.081247 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-28f82c/sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s" Nov 03 00:11:15.057170 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:15.057136 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-28f82c/sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s" Nov 03 00:11:39.884437 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:11:39.884401 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:11:51.296107 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:51.296025 5076 generic.go:356] "Generic (PLEG): container finished" podID="6ae791d6-2d7e-4a26-9904-0b7afb962ff8" containerID="2b52e5d661a97e1ea6a7706f43b26aea9efad61bd78431d3b56fad3b5ea399e7" exitCode=0 Nov 03 00:11:51.296538 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:51.296099 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" event={"ID":"6ae791d6-2d7e-4a26-9904-0b7afb962ff8","Type":"ContainerDied","Data":"2b52e5d661a97e1ea6a7706f43b26aea9efad61bd78431d3b56fad3b5ea399e7"} Nov 03 00:11:51.296538 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:51.296511 5076 scope.go:122] "RemoveContainer" containerID="2b52e5d661a97e1ea6a7706f43b26aea9efad61bd78431d3b56fad3b5ea399e7" Nov 03 00:11:51.298132 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:51.298102 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:11:51.298255 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:51.298185 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:11:51.298255 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:51.298234 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:11:52.304290 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:52.304257 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" event={"ID":"6ae791d6-2d7e-4a26-9904-0b7afb962ff8","Type":"ContainerStarted","Data":"bd45930fc808e7fa71778599884a30f1adb26e4ca92dfaaabf7967e72a0e4bed"} Nov 03 00:11:52.335420 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:52.334945 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" Nov 03 00:11:52.335420 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:52.335272 5076 patch_prober.go:29] interesting pod/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 00:11:52.335420 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:52.335319 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" podUID="6ae791d6-2d7e-4a26-9904-0b7afb962ff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 00:11:53.335020 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:53.328233 5076 patch_prober.go:29] interesting pod/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 00:11:53.335020 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:53.328294 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" podUID="6ae791d6-2d7e-4a26-9904-0b7afb962ff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 00:11:54.326483 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:54.326447 5076 patch_prober.go:29] interesting pod/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 00:11:54.326781 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:54.326754 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" podUID="6ae791d6-2d7e-4a26-9904-0b7afb962ff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 00:11:55.332914 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:55.332294 5076 patch_prober.go:29] interesting pod/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 00:11:55.332914 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:55.332377 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" podUID="6ae791d6-2d7e-4a26-9904-0b7afb962ff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 00:11:56.329914 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:56.329338 5076 patch_prober.go:29] interesting pod/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 00:11:56.329914 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:56.329413 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" podUID="6ae791d6-2d7e-4a26-9904-0b7afb962ff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 00:11:57.326310 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:57.326275 5076 patch_prober.go:29] interesting pod/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 00:11:57.326733 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:57.326344 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" podUID="6ae791d6-2d7e-4a26-9904-0b7afb962ff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 00:11:58.331952 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:58.331921 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" Nov 03 00:11:58.332439 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:58.332286 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" Nov 03 00:11:58.332920 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:11:58.332874 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" Nov 03 00:12:09.883804 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:12:09.883767 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:12:17.686259 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:17.686220 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-59f97a-578d5df469-bmvsq"] Nov 03 00:12:17.787377 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:17.787344 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-59f97a-5c46c69c55-c68jw"] Nov 03 00:12:22.687406 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:22.687336 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-59f97a-578d5df469-bmvsq" podUID="83c7c9f4-adff-43e8-b6f4-0f6abf706b75" containerName="payload-processing" containerID="cri-o://8ef1059f836cf10efb9ff279d589fda9f2f8c047943bfe0f86c3a35c04e6e314" gracePeriod=25 Nov 03 00:12:22.707132 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:22.707092 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-59f97a-578d5df469-bmvsq" podUID="83c7c9f4-adff-43e8-b6f4-0f6abf706b75" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:12:22.788049 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:22.787955 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-59f97a-5c46c69c55-c68jw" podUID="b8ac8868-81dc-42dc-aed6-22e78b580eec" containerName="payload-pre-processing" containerID="cri-o://ffa14c69f3007f0390c69ec5a5b0e74ea90e692c32fbcb186a717385487edc5a" gracePeriod=25 Nov 03 00:12:32.142734 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:32.142706 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 03 00:12:32.235362 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:32.235336 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 03 00:12:32.706451 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:32.706408 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-59f97a-5c46c69c55-c68jw" podUID="b8ac8868-81dc-42dc-aed6-22e78b580eec" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:12:32.706663 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:32.706422 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-59f97a-578d5df469-bmvsq" podUID="83c7c9f4-adff-43e8-b6f4-0f6abf706b75" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:12:39.884311 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:12:39.884269 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:12:42.706404 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:12:42.706379 5076 logging.go:55] [core] [Channel #2997 SubChannel #2999]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.64:50052", ServerName: "10.132.0.64:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.64:50052: connect: connection refused" Nov 03 00:12:42.707182 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:42.707153 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-59f97a-5c46c69c55-c68jw" podUID="b8ac8868-81dc-42dc-aed6-22e78b580eec" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:12:42.965761 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:42.965697 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-59f97a-578d5df469-bmvsq" Nov 03 00:12:43.080873 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:43.080843 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83c7c9f4-adff-43e8-b6f4-0f6abf706b75-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83c7c9f4-adff-43e8-b6f4-0f6abf706b75-plugins-config-volume\") pod \"83c7c9f4-adff-43e8-b6f4-0f6abf706b75\" (UID: \"83c7c9f4-adff-43e8-b6f4-0f6abf706b75\") " Nov 03 00:12:43.081117 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:43.081089 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83c7c9f4-adff-43e8-b6f4-0f6abf706b75-kube-api-access-fpmvj\" (UniqueName: \"kubernetes.io/projected/83c7c9f4-adff-43e8-b6f4-0f6abf706b75-kube-api-access-fpmvj\") pod \"83c7c9f4-adff-43e8-b6f4-0f6abf706b75\" (UID: \"83c7c9f4-adff-43e8-b6f4-0f6abf706b75\") " Nov 03 00:12:43.084452 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:43.081265 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c7c9f4-adff-43e8-b6f4-0f6abf706b75-plugins-config-volume" pod "83c7c9f4-adff-43e8-b6f4-0f6abf706b75" (UID: "83c7c9f4-adff-43e8-b6f4-0f6abf706b75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:12:43.084452 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:43.081609 5076 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83c7c9f4-adff-43e8-b6f4-0f6abf706b75-plugins-config-volume\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:12:43.086799 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:43.086773 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c7c9f4-adff-43e8-b6f4-0f6abf706b75-kube-api-access-fpmvj" pod "83c7c9f4-adff-43e8-b6f4-0f6abf706b75" (UID: "83c7c9f4-adff-43e8-b6f4-0f6abf706b75"). InnerVolumeSpecName "kube-api-access-fpmvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:12:43.090798 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:43.090769 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-59f97a-5c46c69c55-c68jw" Nov 03 00:12:43.182788 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:43.182755 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpmvj\" (UniqueName: \"kubernetes.io/projected/83c7c9f4-adff-43e8-b6f4-0f6abf706b75-kube-api-access-fpmvj\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:12:43.283562 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:43.283460 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8ac8868-81dc-42dc-aed6-22e78b580eec-kube-api-access-xklgt\" (UniqueName: \"kubernetes.io/projected/b8ac8868-81dc-42dc-aed6-22e78b580eec-kube-api-access-xklgt\") pod \"b8ac8868-81dc-42dc-aed6-22e78b580eec\" (UID: \"b8ac8868-81dc-42dc-aed6-22e78b580eec\") " Nov 03 00:12:43.283717 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:43.283569 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8ac8868-81dc-42dc-aed6-22e78b580eec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8ac8868-81dc-42dc-aed6-22e78b580eec-plugins-config-volume\") pod \"b8ac8868-81dc-42dc-aed6-22e78b580eec\" (UID: \"b8ac8868-81dc-42dc-aed6-22e78b580eec\") " Nov 03 00:12:43.285462 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:43.284054 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8ac8868-81dc-42dc-aed6-22e78b580eec-plugins-config-volume" pod "b8ac8868-81dc-42dc-aed6-22e78b580eec" (UID: "b8ac8868-81dc-42dc-aed6-22e78b580eec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:12:43.286032 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:43.286013 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8ac8868-81dc-42dc-aed6-22e78b580eec-kube-api-access-xklgt" pod "b8ac8868-81dc-42dc-aed6-22e78b580eec" (UID: "b8ac8868-81dc-42dc-aed6-22e78b580eec"). InnerVolumeSpecName "kube-api-access-xklgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:12:43.386440 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:43.386402 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xklgt\" (UniqueName: \"kubernetes.io/projected/b8ac8868-81dc-42dc-aed6-22e78b580eec-kube-api-access-xklgt\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:12:43.386440 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:43.386442 5076 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8ac8868-81dc-42dc-aed6-22e78b580eec-plugins-config-volume\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:12:43.685309 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:43.685260 5076 generic.go:356] "Generic (PLEG): container finished" podID="83c7c9f4-adff-43e8-b6f4-0f6abf706b75" containerID="8ef1059f836cf10efb9ff279d589fda9f2f8c047943bfe0f86c3a35c04e6e314" exitCode=0 Nov 03 00:12:43.685510 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:43.685337 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-59f97a-578d5df469-bmvsq" Nov 03 00:12:43.685510 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:43.685335 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-59f97a-578d5df469-bmvsq" event={"ID":"83c7c9f4-adff-43e8-b6f4-0f6abf706b75","Type":"ContainerDied","Data":"8ef1059f836cf10efb9ff279d589fda9f2f8c047943bfe0f86c3a35c04e6e314"} Nov 03 00:12:43.685510 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:43.685463 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-59f97a-578d5df469-bmvsq" event={"ID":"83c7c9f4-adff-43e8-b6f4-0f6abf706b75","Type":"ContainerDied","Data":"d58b78ae2747a576ccbe181246e533e6427dcefd91f8e6d5f6d96bbd1890b6c8"} Nov 03 00:12:43.685510 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:43.685497 5076 scope.go:122] "RemoveContainer" containerID="8ef1059f836cf10efb9ff279d589fda9f2f8c047943bfe0f86c3a35c04e6e314" Nov 03 00:12:43.687349 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:43.686984 5076 generic.go:356] "Generic (PLEG): container finished" podID="b8ac8868-81dc-42dc-aed6-22e78b580eec" containerID="ffa14c69f3007f0390c69ec5a5b0e74ea90e692c32fbcb186a717385487edc5a" exitCode=0 Nov 03 00:12:43.687349 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:43.687030 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-59f97a-5c46c69c55-c68jw" event={"ID":"b8ac8868-81dc-42dc-aed6-22e78b580eec","Type":"ContainerDied","Data":"ffa14c69f3007f0390c69ec5a5b0e74ea90e692c32fbcb186a717385487edc5a"} Nov 03 00:12:43.687349 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:43.687066 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-59f97a-5c46c69c55-c68jw" Nov 03 00:12:43.687349 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:43.687069 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-59f97a-5c46c69c55-c68jw" event={"ID":"b8ac8868-81dc-42dc-aed6-22e78b580eec","Type":"ContainerDied","Data":"1c66c298cfcf46f95a73868af4e0d60cc647afc9a46c1463255cfc0962c30ee6"} Nov 03 00:12:43.696543 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:43.696519 5076 scope.go:122] "RemoveContainer" containerID="8ef1059f836cf10efb9ff279d589fda9f2f8c047943bfe0f86c3a35c04e6e314" Nov 03 00:12:43.696818 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:12:43.696801 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ef1059f836cf10efb9ff279d589fda9f2f8c047943bfe0f86c3a35c04e6e314\": container with ID starting with 8ef1059f836cf10efb9ff279d589fda9f2f8c047943bfe0f86c3a35c04e6e314 not found: ID does not exist" containerID="8ef1059f836cf10efb9ff279d589fda9f2f8c047943bfe0f86c3a35c04e6e314" Nov 03 00:12:43.696862 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:43.696825 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ef1059f836cf10efb9ff279d589fda9f2f8c047943bfe0f86c3a35c04e6e314"} err="failed to get container status \"8ef1059f836cf10efb9ff279d589fda9f2f8c047943bfe0f86c3a35c04e6e314\": rpc error: code = NotFound desc = could not find container \"8ef1059f836cf10efb9ff279d589fda9f2f8c047943bfe0f86c3a35c04e6e314\": container with ID starting with 8ef1059f836cf10efb9ff279d589fda9f2f8c047943bfe0f86c3a35c04e6e314 not found: ID does not exist" Nov 03 00:12:43.696862 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:43.696841 5076 scope.go:122] "RemoveContainer" containerID="ffa14c69f3007f0390c69ec5a5b0e74ea90e692c32fbcb186a717385487edc5a" Nov 03 00:12:43.706373 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:43.706334 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-59f97a-578d5df469-bmvsq" podUID="83c7c9f4-adff-43e8-b6f4-0f6abf706b75" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.64:50052\" within 1s: context deadline exceeded" Nov 03 00:12:43.706508 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:43.706435 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-59f97a-578d5df469-bmvsq" Nov 03 00:12:43.718105 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:43.716724 5076 scope.go:122] "RemoveContainer" containerID="ffa14c69f3007f0390c69ec5a5b0e74ea90e692c32fbcb186a717385487edc5a" Nov 03 00:12:43.718738 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:12:43.718290 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffa14c69f3007f0390c69ec5a5b0e74ea90e692c32fbcb186a717385487edc5a\": container with ID starting with ffa14c69f3007f0390c69ec5a5b0e74ea90e692c32fbcb186a717385487edc5a not found: ID does not exist" containerID="ffa14c69f3007f0390c69ec5a5b0e74ea90e692c32fbcb186a717385487edc5a" Nov 03 00:12:43.718738 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:43.718347 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffa14c69f3007f0390c69ec5a5b0e74ea90e692c32fbcb186a717385487edc5a"} err="failed to get container status \"ffa14c69f3007f0390c69ec5a5b0e74ea90e692c32fbcb186a717385487edc5a\": rpc error: code = NotFound desc = could not find container \"ffa14c69f3007f0390c69ec5a5b0e74ea90e692c32fbcb186a717385487edc5a\": container with ID starting with ffa14c69f3007f0390c69ec5a5b0e74ea90e692c32fbcb186a717385487edc5a not found: ID does not exist" Nov 03 00:12:43.848056 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:43.848019 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-59f97a-578d5df469-bmvsq"] Nov 03 00:12:43.851368 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:43.851343 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-59f97a-578d5df469-bmvsq"] Nov 03 00:12:43.978494 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:43.978402 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-59f97a-5c46c69c55-c68jw"] Nov 03 00:12:43.982144 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:43.982112 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-59f97a-5c46c69c55-c68jw"] Nov 03 00:12:45.148676 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:45.148641 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83c7c9f4-adff-43e8-b6f4-0f6abf706b75" path="/var/lib/kubelet/pods/83c7c9f4-adff-43e8-b6f4-0f6abf706b75/volumes" Nov 03 00:12:45.149103 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:45.149089 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8ac8868-81dc-42dc-aed6-22e78b580eec" path="/var/lib/kubelet/pods/b8ac8868-81dc-42dc-aed6-22e78b580eec/volumes" Nov 03 00:12:47.599253 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:47.599216 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p"] Nov 03 00:12:47.599677 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:47.599409 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p" podUID="3b982eab-939c-49d4-87ec-b6b9a07a3cbc" containerName="maas-api" containerID="cri-o://040852454c25987f1a6d78ad2adccee22518eef86784b376238c4445f6ed5564" gracePeriod=30 Nov 03 00:12:47.714999 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:47.714970 5076 generic.go:356] "Generic (PLEG): container finished" podID="3b982eab-939c-49d4-87ec-b6b9a07a3cbc" containerID="040852454c25987f1a6d78ad2adccee22518eef86784b376238c4445f6ed5564" exitCode=0 Nov 03 00:12:47.715126 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:47.715038 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p" event={"ID":"3b982eab-939c-49d4-87ec-b6b9a07a3cbc","Type":"ContainerDied","Data":"040852454c25987f1a6d78ad2adccee22518eef86784b376238c4445f6ed5564"} Nov 03 00:12:47.917209 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:47.917182 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p" Nov 03 00:12:47.968126 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:47.962181 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b982eab-939c-49d4-87ec-b6b9a07a3cbc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b982eab-939c-49d4-87ec-b6b9a07a3cbc-maas-api-tls\") pod \"3b982eab-939c-49d4-87ec-b6b9a07a3cbc\" (UID: \"3b982eab-939c-49d4-87ec-b6b9a07a3cbc\") " Nov 03 00:12:47.968126 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:47.962784 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b982eab-939c-49d4-87ec-b6b9a07a3cbc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b982eab-939c-49d4-87ec-b6b9a07a3cbc-metrics-certs\") pod \"3b982eab-939c-49d4-87ec-b6b9a07a3cbc\" (UID: \"3b982eab-939c-49d4-87ec-b6b9a07a3cbc\") " Nov 03 00:12:47.968126 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:47.964296 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b982eab-939c-49d4-87ec-b6b9a07a3cbc-kube-api-access-djgrw\" (UniqueName: \"kubernetes.io/projected/3b982eab-939c-49d4-87ec-b6b9a07a3cbc-kube-api-access-djgrw\") pod \"3b982eab-939c-49d4-87ec-b6b9a07a3cbc\" (UID: \"3b982eab-939c-49d4-87ec-b6b9a07a3cbc\") " Nov 03 00:12:47.968126 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:47.964362 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b982eab-939c-49d4-87ec-b6b9a07a3cbc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b982eab-939c-49d4-87ec-b6b9a07a3cbc-openshift-service-ca\") pod \"3b982eab-939c-49d4-87ec-b6b9a07a3cbc\" (UID: \"3b982eab-939c-49d4-87ec-b6b9a07a3cbc\") " Nov 03 00:12:47.968126 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:47.965297 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b982eab-939c-49d4-87ec-b6b9a07a3cbc-openshift-service-ca" pod "3b982eab-939c-49d4-87ec-b6b9a07a3cbc" (UID: "3b982eab-939c-49d4-87ec-b6b9a07a3cbc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:12:47.968126 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:47.966147 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b982eab-939c-49d4-87ec-b6b9a07a3cbc-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:12:47.981942 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:47.972959 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b982eab-939c-49d4-87ec-b6b9a07a3cbc-maas-api-tls" pod "3b982eab-939c-49d4-87ec-b6b9a07a3cbc" (UID: "3b982eab-939c-49d4-87ec-b6b9a07a3cbc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:12:47.981942 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:47.974991 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b982eab-939c-49d4-87ec-b6b9a07a3cbc-metrics-certs" pod "3b982eab-939c-49d4-87ec-b6b9a07a3cbc" (UID: "3b982eab-939c-49d4-87ec-b6b9a07a3cbc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:12:47.997535 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:47.986607 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b982eab-939c-49d4-87ec-b6b9a07a3cbc-kube-api-access-djgrw" pod "3b982eab-939c-49d4-87ec-b6b9a07a3cbc" (UID: "3b982eab-939c-49d4-87ec-b6b9a07a3cbc"). InnerVolumeSpecName "kube-api-access-djgrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:12:48.067616 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:48.067584 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b982eab-939c-49d4-87ec-b6b9a07a3cbc-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:12:48.067616 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:48.067613 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djgrw\" (UniqueName: \"kubernetes.io/projected/3b982eab-939c-49d4-87ec-b6b9a07a3cbc-kube-api-access-djgrw\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:12:48.067616 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:48.067624 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b982eab-939c-49d4-87ec-b6b9a07a3cbc-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:12:48.723581 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:48.723548 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p" event={"ID":"3b982eab-939c-49d4-87ec-b6b9a07a3cbc","Type":"ContainerDied","Data":"6428135c69adfa69aabf46b146a3ae4bdee8456958674b6ed65dc90d435b11ca"} Nov 03 00:12:48.723581 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:48.723580 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p" Nov 03 00:12:48.724104 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:48.723610 5076 scope.go:122] "RemoveContainer" containerID="040852454c25987f1a6d78ad2adccee22518eef86784b376238c4445f6ed5564" Nov 03 00:12:48.880044 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:48.880004 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p"] Nov 03 00:12:48.927557 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:48.927516 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-59f97a-6b4d4d9d9c-hm89p"] Nov 03 00:12:49.147967 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:49.147876 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b982eab-939c-49d4-87ec-b6b9a07a3cbc" path="/var/lib/kubelet/pods/3b982eab-939c-49d4-87ec-b6b9a07a3cbc/volumes" Nov 03 00:12:50.148905 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.148859 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj"] Nov 03 00:12:50.149428 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.149413 5076 cpu_manager.go:414] "containerMap: removing container" podUID="83c7c9f4-adff-43e8-b6f4-0f6abf706b75" containerName="payload-processing" Nov 03 00:12:50.149474 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.149428 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83c7c9f4-adff-43e8-b6f4-0f6abf706b75" containerName="payload-processing" Nov 03 00:12:50.149474 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.149443 5076 cpu_manager.go:414] "containerMap: removing container" podUID="3b982eab-939c-49d4-87ec-b6b9a07a3cbc" containerName="maas-api" Nov 03 00:12:50.149474 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.149448 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b982eab-939c-49d4-87ec-b6b9a07a3cbc" containerName="maas-api" Nov 03 00:12:50.149573 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.149482 5076 cpu_manager.go:414] "containerMap: removing container" podUID="b8ac8868-81dc-42dc-aed6-22e78b580eec" containerName="payload-pre-processing" Nov 03 00:12:50.149573 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.149488 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8ac8868-81dc-42dc-aed6-22e78b580eec" containerName="payload-pre-processing" Nov 03 00:12:50.149638 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.149579 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="83c7c9f4-adff-43e8-b6f4-0f6abf706b75" containerName="payload-processing" Nov 03 00:12:50.149638 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.149587 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b982eab-939c-49d4-87ec-b6b9a07a3cbc" containerName="maas-api" Nov 03 00:12:50.149638 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.149601 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8ac8868-81dc-42dc-aed6-22e78b580eec" containerName="payload-pre-processing" Nov 03 00:12:50.153074 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.153050 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" Nov 03 00:12:50.213517 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.213482 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d78fb30-3ea3-45c8-9761-31fe6b941839-istiod-ca-cert\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj\" (UID: \"0d78fb30-3ea3-45c8-9761-31fe6b941839\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" Nov 03 00:12:50.213699 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.213527 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d78fb30-3ea3-45c8-9761-31fe6b941839-istio-envoy\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj\" (UID: \"0d78fb30-3ea3-45c8-9761-31fe6b941839\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" Nov 03 00:12:50.213699 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.213554 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d78fb30-3ea3-45c8-9761-31fe6b941839-istio-token\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj\" (UID: \"0d78fb30-3ea3-45c8-9761-31fe6b941839\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" Nov 03 00:12:50.213778 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.213692 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d78fb30-3ea3-45c8-9761-31fe6b941839-istio-podinfo\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj\" (UID: \"0d78fb30-3ea3-45c8-9761-31fe6b941839\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" Nov 03 00:12:50.213838 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.213821 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl7l5\" (UniqueName: \"kubernetes.io/projected/0d78fb30-3ea3-45c8-9761-31fe6b941839-kube-api-access-cl7l5\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj\" (UID: \"0d78fb30-3ea3-45c8-9761-31fe6b941839\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" Nov 03 00:12:50.213880 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.213866 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d78fb30-3ea3-45c8-9761-31fe6b941839-workload-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj\" (UID: \"0d78fb30-3ea3-45c8-9761-31fe6b941839\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" Nov 03 00:12:50.214085 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.214063 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d78fb30-3ea3-45c8-9761-31fe6b941839-workload-certs\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj\" (UID: \"0d78fb30-3ea3-45c8-9761-31fe6b941839\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" Nov 03 00:12:50.214190 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.214164 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d78fb30-3ea3-45c8-9761-31fe6b941839-credential-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj\" (UID: \"0d78fb30-3ea3-45c8-9761-31fe6b941839\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" Nov 03 00:12:50.214332 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.214267 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d78fb30-3ea3-45c8-9761-31fe6b941839-istio-data\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj\" (UID: \"0d78fb30-3ea3-45c8-9761-31fe6b941839\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" Nov 03 00:12:50.250538 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.250486 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-59f97a-openshift-default-dockercfg-r7nkq\"" Nov 03 00:12:50.255151 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.255077 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj"] Nov 03 00:12:50.315126 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.315092 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d78fb30-3ea3-45c8-9761-31fe6b941839-workload-certs\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj\" (UID: \"0d78fb30-3ea3-45c8-9761-31fe6b941839\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" Nov 03 00:12:50.315342 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.315148 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d78fb30-3ea3-45c8-9761-31fe6b941839-credential-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj\" (UID: \"0d78fb30-3ea3-45c8-9761-31fe6b941839\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" Nov 03 00:12:50.315342 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.315192 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d78fb30-3ea3-45c8-9761-31fe6b941839-istio-data\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj\" (UID: \"0d78fb30-3ea3-45c8-9761-31fe6b941839\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" Nov 03 00:12:50.315342 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.315221 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d78fb30-3ea3-45c8-9761-31fe6b941839-istiod-ca-cert\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj\" (UID: \"0d78fb30-3ea3-45c8-9761-31fe6b941839\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" Nov 03 00:12:50.315342 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.315243 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d78fb30-3ea3-45c8-9761-31fe6b941839-istio-envoy\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj\" (UID: \"0d78fb30-3ea3-45c8-9761-31fe6b941839\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" Nov 03 00:12:50.315592 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.315378 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d78fb30-3ea3-45c8-9761-31fe6b941839-istio-token\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj\" (UID: \"0d78fb30-3ea3-45c8-9761-31fe6b941839\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" Nov 03 00:12:50.315592 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.315447 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d78fb30-3ea3-45c8-9761-31fe6b941839-istio-podinfo\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj\" (UID: \"0d78fb30-3ea3-45c8-9761-31fe6b941839\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" Nov 03 00:12:50.315592 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.315537 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d78fb30-3ea3-45c8-9761-31fe6b941839-credential-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj\" (UID: \"0d78fb30-3ea3-45c8-9761-31fe6b941839\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" Nov 03 00:12:50.315592 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.315559 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl7l5\" (UniqueName: \"kubernetes.io/projected/0d78fb30-3ea3-45c8-9761-31fe6b941839-kube-api-access-cl7l5\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj\" (UID: \"0d78fb30-3ea3-45c8-9761-31fe6b941839\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" Nov 03 00:12:50.315786 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.315599 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d78fb30-3ea3-45c8-9761-31fe6b941839-workload-certs\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj\" (UID: \"0d78fb30-3ea3-45c8-9761-31fe6b941839\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" Nov 03 00:12:50.315786 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.315620 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d78fb30-3ea3-45c8-9761-31fe6b941839-workload-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj\" (UID: \"0d78fb30-3ea3-45c8-9761-31fe6b941839\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" Nov 03 00:12:50.315786 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.315669 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d78fb30-3ea3-45c8-9761-31fe6b941839-istio-data\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj\" (UID: \"0d78fb30-3ea3-45c8-9761-31fe6b941839\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" Nov 03 00:12:50.315929 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.315913 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d78fb30-3ea3-45c8-9761-31fe6b941839-workload-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj\" (UID: \"0d78fb30-3ea3-45c8-9761-31fe6b941839\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" Nov 03 00:12:50.316034 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.316012 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d78fb30-3ea3-45c8-9761-31fe6b941839-istiod-ca-cert\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj\" (UID: \"0d78fb30-3ea3-45c8-9761-31fe6b941839\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" Nov 03 00:12:50.317880 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.317859 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d78fb30-3ea3-45c8-9761-31fe6b941839-istio-envoy\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj\" (UID: \"0d78fb30-3ea3-45c8-9761-31fe6b941839\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" Nov 03 00:12:50.318071 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.318052 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d78fb30-3ea3-45c8-9761-31fe6b941839-istio-podinfo\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj\" (UID: \"0d78fb30-3ea3-45c8-9761-31fe6b941839\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" Nov 03 00:12:50.370308 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.370270 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d78fb30-3ea3-45c8-9761-31fe6b941839-istio-token\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj\" (UID: \"0d78fb30-3ea3-45c8-9761-31fe6b941839\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" Nov 03 00:12:50.370465 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.370423 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl7l5\" (UniqueName: \"kubernetes.io/projected/0d78fb30-3ea3-45c8-9761-31fe6b941839-kube-api-access-cl7l5\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj\" (UID: \"0d78fb30-3ea3-45c8-9761-31fe6b941839\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" Nov 03 00:12:50.467466 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.467430 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" Nov 03 00:12:50.885168 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.885132 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj"] Nov 03 00:12:50.885990 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:12:50.885957 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d78fb30_3ea3_45c8_9761_31fe6b941839.slice/crio-0ac5276695baf16c6f5c557128e31e6144e9632e528cd960b43bdc6ac6f62747 WatchSource:0}: Error finding container 0ac5276695baf16c6f5c557128e31e6144e9632e528cd960b43bdc6ac6f62747: Status 404 returned error can't find the container with id 0ac5276695baf16c6f5c557128e31e6144e9632e528cd960b43bdc6ac6f62747 Nov 03 00:12:50.888588 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.888555 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:12:50.888657 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.888624 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:12:50.888696 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:50.888656 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:12:51.742697 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:51.742655 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" event={"ID":"0d78fb30-3ea3-45c8-9761-31fe6b941839","Type":"ContainerStarted","Data":"4f980c321a395a4516224274175dfbd836899f66e49e20364ac630c8327770f8"} Nov 03 00:12:51.743188 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:51.742722 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" event={"ID":"0d78fb30-3ea3-45c8-9761-31fe6b941839","Type":"ContainerStarted","Data":"0ac5276695baf16c6f5c557128e31e6144e9632e528cd960b43bdc6ac6f62747"} Nov 03 00:12:51.970868 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:51.970821 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" podStartSLOduration=1.970807035 podStartE2EDuration="1.970807035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:12:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:12:51.923371805 +0000 UTC m=+4820.891903590" watchObservedRunningTime="2026-11-03 00:12:51.970807035 +0000 UTC m=+4820.939338813" Nov 03 00:12:52.468032 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:52.467993 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" Nov 03 00:12:52.470031 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:52.470003 5076 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 03 00:12:52.470150 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:52.470050 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" podUID="0d78fb30-3ea3-45c8-9761-31fe6b941839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 03 00:12:53.469373 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:53.469332 5076 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 03 00:12:53.470013 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:53.469976 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" podUID="0d78fb30-3ea3-45c8-9761-31fe6b941839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 03 00:12:54.138610 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:54.138570 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj"] Nov 03 00:12:54.469318 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:54.469280 5076 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 03 00:12:54.469688 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:54.469663 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" podUID="0d78fb30-3ea3-45c8-9761-31fe6b941839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 03 00:12:54.762426 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:12:54.762312 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" podUID="0d78fb30-3ea3-45c8-9761-31fe6b941839" containerName="istio-proxy" containerID="cri-o://4f980c321a395a4516224274175dfbd836899f66e49e20364ac630c8327770f8" gracePeriod=30 Nov 03 00:13:03.681397 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:13:03.681370 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" Nov 03 00:13:03.817715 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:13:03.817680 5076 generic.go:356] "Generic (PLEG): container finished" podID="0d78fb30-3ea3-45c8-9761-31fe6b941839" containerID="4f980c321a395a4516224274175dfbd836899f66e49e20364ac630c8327770f8" exitCode=0 Nov 03 00:13:03.817908 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:13:03.817724 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" Nov 03 00:13:03.817908 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:13:03.817767 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" event={"ID":"0d78fb30-3ea3-45c8-9761-31fe6b941839","Type":"ContainerDied","Data":"4f980c321a395a4516224274175dfbd836899f66e49e20364ac630c8327770f8"} Nov 03 00:13:03.817908 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:13:03.817825 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj" event={"ID":"0d78fb30-3ea3-45c8-9761-31fe6b941839","Type":"ContainerDied","Data":"0ac5276695baf16c6f5c557128e31e6144e9632e528cd960b43bdc6ac6f62747"} Nov 03 00:13:03.817908 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:13:03.817852 5076 scope.go:122] "RemoveContainer" containerID="4f980c321a395a4516224274175dfbd836899f66e49e20364ac630c8327770f8" Nov 03 00:13:03.827956 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:13:03.827938 5076 scope.go:122] "RemoveContainer" containerID="4f980c321a395a4516224274175dfbd836899f66e49e20364ac630c8327770f8" Nov 03 00:13:03.828239 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:13:03.828215 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f980c321a395a4516224274175dfbd836899f66e49e20364ac630c8327770f8\": container with ID starting with 4f980c321a395a4516224274175dfbd836899f66e49e20364ac630c8327770f8 not found: ID does not exist" containerID="4f980c321a395a4516224274175dfbd836899f66e49e20364ac630c8327770f8" Nov 03 00:13:03.828318 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:13:03.828250 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f980c321a395a4516224274175dfbd836899f66e49e20364ac630c8327770f8"} err="failed to get container status \"4f980c321a395a4516224274175dfbd836899f66e49e20364ac630c8327770f8\": rpc error: code = NotFound desc = could not find container \"4f980c321a395a4516224274175dfbd836899f66e49e20364ac630c8327770f8\": container with ID starting with 4f980c321a395a4516224274175dfbd836899f66e49e20364ac630c8327770f8 not found: ID does not exist" Nov 03 00:13:03.835765 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:13:03.835711 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d78fb30-3ea3-45c8-9761-31fe6b941839-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d78fb30-3ea3-45c8-9761-31fe6b941839-istio-envoy\") pod \"0d78fb30-3ea3-45c8-9761-31fe6b941839\" (UID: \"0d78fb30-3ea3-45c8-9761-31fe6b941839\") " Nov 03 00:13:03.836571 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:13:03.836012 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d78fb30-3ea3-45c8-9761-31fe6b941839-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d78fb30-3ea3-45c8-9761-31fe6b941839-istio-podinfo\") pod \"0d78fb30-3ea3-45c8-9761-31fe6b941839\" (UID: \"0d78fb30-3ea3-45c8-9761-31fe6b941839\") " Nov 03 00:13:03.836571 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:13:03.836164 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d78fb30-3ea3-45c8-9761-31fe6b941839-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d78fb30-3ea3-45c8-9761-31fe6b941839-istiod-ca-cert\") pod \"0d78fb30-3ea3-45c8-9761-31fe6b941839\" (UID: \"0d78fb30-3ea3-45c8-9761-31fe6b941839\") " Nov 03 00:13:03.836571 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:13:03.836199 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d78fb30-3ea3-45c8-9761-31fe6b941839-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d78fb30-3ea3-45c8-9761-31fe6b941839-credential-socket\") pod \"0d78fb30-3ea3-45c8-9761-31fe6b941839\" (UID: \"0d78fb30-3ea3-45c8-9761-31fe6b941839\") " Nov 03 00:13:03.836571 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:13:03.836230 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d78fb30-3ea3-45c8-9761-31fe6b941839-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d78fb30-3ea3-45c8-9761-31fe6b941839-workload-certs\") pod \"0d78fb30-3ea3-45c8-9761-31fe6b941839\" (UID: \"0d78fb30-3ea3-45c8-9761-31fe6b941839\") " Nov 03 00:13:03.836571 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:13:03.836256 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d78fb30-3ea3-45c8-9761-31fe6b941839-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d78fb30-3ea3-45c8-9761-31fe6b941839-istio-data\") pod \"0d78fb30-3ea3-45c8-9761-31fe6b941839\" (UID: \"0d78fb30-3ea3-45c8-9761-31fe6b941839\") " Nov 03 00:13:03.836571 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:13:03.836290 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d78fb30-3ea3-45c8-9761-31fe6b941839-kube-api-access-cl7l5\" (UniqueName: \"kubernetes.io/projected/0d78fb30-3ea3-45c8-9761-31fe6b941839-kube-api-access-cl7l5\") pod \"0d78fb30-3ea3-45c8-9761-31fe6b941839\" (UID: \"0d78fb30-3ea3-45c8-9761-31fe6b941839\") " Nov 03 00:13:03.836571 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:13:03.836412 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d78fb30-3ea3-45c8-9761-31fe6b941839-istio-token\" (UniqueName: \"kubernetes.io/projected/0d78fb30-3ea3-45c8-9761-31fe6b941839-istio-token\") pod \"0d78fb30-3ea3-45c8-9761-31fe6b941839\" (UID: \"0d78fb30-3ea3-45c8-9761-31fe6b941839\") " Nov 03 00:13:03.836571 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:13:03.836521 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d78fb30-3ea3-45c8-9761-31fe6b941839-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d78fb30-3ea3-45c8-9761-31fe6b941839-workload-socket\") pod \"0d78fb30-3ea3-45c8-9761-31fe6b941839\" (UID: \"0d78fb30-3ea3-45c8-9761-31fe6b941839\") " Nov 03 00:13:03.837343 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:13:03.837297 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d78fb30-3ea3-45c8-9761-31fe6b941839-credential-socket" pod "0d78fb30-3ea3-45c8-9761-31fe6b941839" (UID: "0d78fb30-3ea3-45c8-9761-31fe6b941839"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:13:03.837343 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:13:03.837314 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d78fb30-3ea3-45c8-9761-31fe6b941839-workload-certs" pod "0d78fb30-3ea3-45c8-9761-31fe6b941839" (UID: "0d78fb30-3ea3-45c8-9761-31fe6b941839"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:13:03.837502 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:13:03.837375 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d78fb30-3ea3-45c8-9761-31fe6b941839-istio-data" pod "0d78fb30-3ea3-45c8-9761-31fe6b941839" (UID: "0d78fb30-3ea3-45c8-9761-31fe6b941839"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:13:03.837835 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:13:03.837810 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d78fb30-3ea3-45c8-9761-31fe6b941839-istiod-ca-cert" pod "0d78fb30-3ea3-45c8-9761-31fe6b941839" (UID: "0d78fb30-3ea3-45c8-9761-31fe6b941839"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:13:03.838083 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:13:03.837981 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d78fb30-3ea3-45c8-9761-31fe6b941839-workload-socket" pod "0d78fb30-3ea3-45c8-9761-31fe6b941839" (UID: "0d78fb30-3ea3-45c8-9761-31fe6b941839"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:13:03.838817 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:13:03.838795 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d78fb30-3ea3-45c8-9761-31fe6b941839-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:13:03.839062 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:13:03.839044 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d78fb30-3ea3-45c8-9761-31fe6b941839-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:13:03.839231 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:13:03.839216 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d78fb30-3ea3-45c8-9761-31fe6b941839-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:13:03.839371 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:13:03.839356 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d78fb30-3ea3-45c8-9761-31fe6b941839-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:13:03.839504 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:13:03.839487 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d78fb30-3ea3-45c8-9761-31fe6b941839-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:13:03.840904 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:13:03.840868 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d78fb30-3ea3-45c8-9761-31fe6b941839-kube-api-access-cl7l5" pod "0d78fb30-3ea3-45c8-9761-31fe6b941839" (UID: "0d78fb30-3ea3-45c8-9761-31fe6b941839"). InnerVolumeSpecName "kube-api-access-cl7l5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:13:03.841105 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:13:03.841078 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d78fb30-3ea3-45c8-9761-31fe6b941839-istio-podinfo" pod "0d78fb30-3ea3-45c8-9761-31fe6b941839" (UID: "0d78fb30-3ea3-45c8-9761-31fe6b941839"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:13:03.843175 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:13:03.843151 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d78fb30-3ea3-45c8-9761-31fe6b941839-istio-token" pod "0d78fb30-3ea3-45c8-9761-31fe6b941839" (UID: "0d78fb30-3ea3-45c8-9761-31fe6b941839"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:13:03.843281 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:13:03.843220 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d78fb30-3ea3-45c8-9761-31fe6b941839-istio-envoy" pod "0d78fb30-3ea3-45c8-9761-31fe6b941839" (UID: "0d78fb30-3ea3-45c8-9761-31fe6b941839"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:13:03.940623 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:13:03.940581 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d78fb30-3ea3-45c8-9761-31fe6b941839-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:13:03.940623 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:13:03.940616 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl7l5\" (UniqueName: \"kubernetes.io/projected/0d78fb30-3ea3-45c8-9761-31fe6b941839-kube-api-access-cl7l5\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:13:03.940843 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:13:03.940634 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d78fb30-3ea3-45c8-9761-31fe6b941839-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:13:03.940843 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:13:03.940647 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d78fb30-3ea3-45c8-9761-31fe6b941839-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:13:04.234748 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:13:04.234707 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj"] Nov 03 00:13:04.282308 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:13:04.282266 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-r4qnj"] Nov 03 00:13:05.148195 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:13:05.148161 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d78fb30-3ea3-45c8-9761-31fe6b941839" path="/var/lib/kubelet/pods/0d78fb30-3ea3-45c8-9761-31fe6b941839/volumes" Nov 03 00:13:09.883207 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:13:09.883171 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:13:39.923819 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:13:39.923776 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:14:09.889653 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:14:09.889612 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:14:30.953934 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:30.953885 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx"] Nov 03 00:14:30.954333 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:30.954127 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" podUID="905534e2-1b25-484c-9030-0185f3148cec" containerName="istio-proxy" containerID="cri-o://28c264b6df8b858f92ca502b53065b4f85ed636dcb35fd51846eeee5ccf7e7dc" gracePeriod=30 Nov 03 00:14:31.003074 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:31.003034 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p"] Nov 03 00:14:31.003369 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:31.003341 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" podUID="b45216a5-f1ba-4744-bd22-c59437513e86" containerName="istio-proxy" containerID="cri-o://de90b89e5e6cd36d6e364b767496267f914aeb7089b9aa8ce383c4453f1a5d07" gracePeriod=30 Nov 03 00:14:31.007301 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:31.007276 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh"] Nov 03 00:14:31.007539 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:31.007514 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" podUID="5c9698d7-4d4b-44e6-8165-106b31bcae2b" containerName="istio-proxy" containerID="cri-o://a95800e962c583503e5aa7bd4c2d76e57a10d386554236ab6b8eec952348c535" gracePeriod=30 Nov 03 00:14:34.048968 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:34.048927 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-89gnx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:14:34.049388 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:34.048999 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" podUID="905534e2-1b25-484c-9030-0185f3148cec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:14:35.100749 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:35.100697 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-66nkh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:14:35.101191 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:35.100781 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" podUID="5c9698d7-4d4b-44e6-8165-106b31bcae2b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:14:36.077098 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.077068 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-vx88p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 03 00:14:36.077251 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.077133 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" podUID="b45216a5-f1ba-4744-bd22-c59437513e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 03 00:14:36.289919 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.289616 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" Nov 03 00:14:36.341397 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.341372 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" Nov 03 00:14:36.344979 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.344954 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" Nov 03 00:14:36.387316 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.387283 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/905534e2-1b25-484c-9030-0185f3148cec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/905534e2-1b25-484c-9030-0185f3148cec-workload-certs\") pod \"905534e2-1b25-484c-9030-0185f3148cec\" (UID: \"905534e2-1b25-484c-9030-0185f3148cec\") " Nov 03 00:14:36.387510 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.387336 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/905534e2-1b25-484c-9030-0185f3148cec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/905534e2-1b25-484c-9030-0185f3148cec-credential-socket\") pod \"905534e2-1b25-484c-9030-0185f3148cec\" (UID: \"905534e2-1b25-484c-9030-0185f3148cec\") " Nov 03 00:14:36.387510 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.387385 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/905534e2-1b25-484c-9030-0185f3148cec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/905534e2-1b25-484c-9030-0185f3148cec-istio-data\") pod \"905534e2-1b25-484c-9030-0185f3148cec\" (UID: \"905534e2-1b25-484c-9030-0185f3148cec\") " Nov 03 00:14:36.387647 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.387550 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/905534e2-1b25-484c-9030-0185f3148cec-workload-certs" pod "905534e2-1b25-484c-9030-0185f3148cec" (UID: "905534e2-1b25-484c-9030-0185f3148cec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:36.387647 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.387554 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/905534e2-1b25-484c-9030-0185f3148cec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/905534e2-1b25-484c-9030-0185f3148cec-workload-socket\") pod \"905534e2-1b25-484c-9030-0185f3148cec\" (UID: \"905534e2-1b25-484c-9030-0185f3148cec\") " Nov 03 00:14:36.387647 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.387633 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/905534e2-1b25-484c-9030-0185f3148cec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/905534e2-1b25-484c-9030-0185f3148cec-istio-envoy\") pod \"905534e2-1b25-484c-9030-0185f3148cec\" (UID: \"905534e2-1b25-484c-9030-0185f3148cec\") " Nov 03 00:14:36.387803 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.387653 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/905534e2-1b25-484c-9030-0185f3148cec-credential-socket" pod "905534e2-1b25-484c-9030-0185f3148cec" (UID: "905534e2-1b25-484c-9030-0185f3148cec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:36.387803 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.387713 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/905534e2-1b25-484c-9030-0185f3148cec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/905534e2-1b25-484c-9030-0185f3148cec-istiod-ca-cert\") pod \"905534e2-1b25-484c-9030-0185f3148cec\" (UID: \"905534e2-1b25-484c-9030-0185f3148cec\") " Nov 03 00:14:36.387803 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.387740 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/905534e2-1b25-484c-9030-0185f3148cec-istio-token\" (UniqueName: \"kubernetes.io/projected/905534e2-1b25-484c-9030-0185f3148cec-istio-token\") pod \"905534e2-1b25-484c-9030-0185f3148cec\" (UID: \"905534e2-1b25-484c-9030-0185f3148cec\") " Nov 03 00:14:36.387803 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.387777 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/905534e2-1b25-484c-9030-0185f3148cec-workload-socket" pod "905534e2-1b25-484c-9030-0185f3148cec" (UID: "905534e2-1b25-484c-9030-0185f3148cec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:36.388048 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.387843 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/905534e2-1b25-484c-9030-0185f3148cec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/905534e2-1b25-484c-9030-0185f3148cec-istio-podinfo\") pod \"905534e2-1b25-484c-9030-0185f3148cec\" (UID: \"905534e2-1b25-484c-9030-0185f3148cec\") " Nov 03 00:14:36.388048 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.387916 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/905534e2-1b25-484c-9030-0185f3148cec-kube-api-access-5l56l\" (UniqueName: \"kubernetes.io/projected/905534e2-1b25-484c-9030-0185f3148cec-kube-api-access-5l56l\") pod \"905534e2-1b25-484c-9030-0185f3148cec\" (UID: \"905534e2-1b25-484c-9030-0185f3148cec\") " Nov 03 00:14:36.388207 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.388183 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/905534e2-1b25-484c-9030-0185f3148cec-istiod-ca-cert" pod "905534e2-1b25-484c-9030-0185f3148cec" (UID: "905534e2-1b25-484c-9030-0185f3148cec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:14:36.388334 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.388259 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/905534e2-1b25-484c-9030-0185f3148cec-istio-data" pod "905534e2-1b25-484c-9030-0185f3148cec" (UID: "905534e2-1b25-484c-9030-0185f3148cec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:36.388529 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.388508 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/905534e2-1b25-484c-9030-0185f3148cec-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:36.388620 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.388533 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/905534e2-1b25-484c-9030-0185f3148cec-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:36.388620 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.388549 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/905534e2-1b25-484c-9030-0185f3148cec-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:36.388620 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.388563 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/905534e2-1b25-484c-9030-0185f3148cec-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:36.388620 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.388579 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/905534e2-1b25-484c-9030-0185f3148cec-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:36.389557 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.389534 5076 generic.go:356] "Generic (PLEG): container finished" podID="905534e2-1b25-484c-9030-0185f3148cec" containerID="28c264b6df8b858f92ca502b53065b4f85ed636dcb35fd51846eeee5ccf7e7dc" exitCode=0 Nov 03 00:14:36.389654 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.389605 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" Nov 03 00:14:36.389654 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.389623 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" event={"ID":"905534e2-1b25-484c-9030-0185f3148cec","Type":"ContainerDied","Data":"28c264b6df8b858f92ca502b53065b4f85ed636dcb35fd51846eeee5ccf7e7dc"} Nov 03 00:14:36.389789 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.389691 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx" event={"ID":"905534e2-1b25-484c-9030-0185f3148cec","Type":"ContainerDied","Data":"6e93fa98d475eea1dfbc955df4c7607bca17c60b6e224934c4336e090a7bc9e4"} Nov 03 00:14:36.389789 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.389728 5076 scope.go:122] "RemoveContainer" containerID="28c264b6df8b858f92ca502b53065b4f85ed636dcb35fd51846eeee5ccf7e7dc" Nov 03 00:14:36.390282 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.390260 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/905534e2-1b25-484c-9030-0185f3148cec-istio-token" pod "905534e2-1b25-484c-9030-0185f3148cec" (UID: "905534e2-1b25-484c-9030-0185f3148cec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:14:36.390372 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.390304 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/905534e2-1b25-484c-9030-0185f3148cec-istio-envoy" pod "905534e2-1b25-484c-9030-0185f3148cec" (UID: "905534e2-1b25-484c-9030-0185f3148cec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:36.390545 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.390521 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/905534e2-1b25-484c-9030-0185f3148cec-kube-api-access-5l56l" pod "905534e2-1b25-484c-9030-0185f3148cec" (UID: "905534e2-1b25-484c-9030-0185f3148cec"). InnerVolumeSpecName "kube-api-access-5l56l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:14:36.390644 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.390576 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/905534e2-1b25-484c-9030-0185f3148cec-istio-podinfo" pod "905534e2-1b25-484c-9030-0185f3148cec" (UID: "905534e2-1b25-484c-9030-0185f3148cec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:14:36.391439 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.391418 5076 generic.go:356] "Generic (PLEG): container finished" podID="5c9698d7-4d4b-44e6-8165-106b31bcae2b" containerID="a95800e962c583503e5aa7bd4c2d76e57a10d386554236ab6b8eec952348c535" exitCode=0 Nov 03 00:14:36.391529 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.391478 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" Nov 03 00:14:36.391529 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.391492 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" event={"ID":"5c9698d7-4d4b-44e6-8165-106b31bcae2b","Type":"ContainerDied","Data":"a95800e962c583503e5aa7bd4c2d76e57a10d386554236ab6b8eec952348c535"} Nov 03 00:14:36.391632 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.391529 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh" event={"ID":"5c9698d7-4d4b-44e6-8165-106b31bcae2b","Type":"ContainerDied","Data":"d971080f0d99064aae83ced42e09ca4cd9ea6b78d65bc93cf77b6327069fe38e"} Nov 03 00:14:36.392927 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.392904 5076 generic.go:356] "Generic (PLEG): container finished" podID="b45216a5-f1ba-4744-bd22-c59437513e86" containerID="de90b89e5e6cd36d6e364b767496267f914aeb7089b9aa8ce383c4453f1a5d07" exitCode=0 Nov 03 00:14:36.393029 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.392932 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" event={"ID":"b45216a5-f1ba-4744-bd22-c59437513e86","Type":"ContainerDied","Data":"de90b89e5e6cd36d6e364b767496267f914aeb7089b9aa8ce383c4453f1a5d07"} Nov 03 00:14:36.393029 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.392995 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" event={"ID":"b45216a5-f1ba-4744-bd22-c59437513e86","Type":"ContainerDied","Data":"7d8b14eb9ecf7a25de77bc9439afb588c0e36230a0f1eace3d8ad482b46823bf"} Nov 03 00:14:36.393029 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.392946 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p" Nov 03 00:14:36.402440 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.401867 5076 scope.go:122] "RemoveContainer" containerID="28c264b6df8b858f92ca502b53065b4f85ed636dcb35fd51846eeee5ccf7e7dc" Nov 03 00:14:36.402851 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:14:36.402727 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28c264b6df8b858f92ca502b53065b4f85ed636dcb35fd51846eeee5ccf7e7dc\": container with ID starting with 28c264b6df8b858f92ca502b53065b4f85ed636dcb35fd51846eeee5ccf7e7dc not found: ID does not exist" containerID="28c264b6df8b858f92ca502b53065b4f85ed636dcb35fd51846eeee5ccf7e7dc" Nov 03 00:14:36.402851 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.402763 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28c264b6df8b858f92ca502b53065b4f85ed636dcb35fd51846eeee5ccf7e7dc"} err="failed to get container status \"28c264b6df8b858f92ca502b53065b4f85ed636dcb35fd51846eeee5ccf7e7dc\": rpc error: code = NotFound desc = could not find container \"28c264b6df8b858f92ca502b53065b4f85ed636dcb35fd51846eeee5ccf7e7dc\": container with ID starting with 28c264b6df8b858f92ca502b53065b4f85ed636dcb35fd51846eeee5ccf7e7dc not found: ID does not exist" Nov 03 00:14:36.402851 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.402785 5076 scope.go:122] "RemoveContainer" containerID="a95800e962c583503e5aa7bd4c2d76e57a10d386554236ab6b8eec952348c535" Nov 03 00:14:36.411843 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.411826 5076 scope.go:122] "RemoveContainer" containerID="a95800e962c583503e5aa7bd4c2d76e57a10d386554236ab6b8eec952348c535" Nov 03 00:14:36.412172 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:14:36.412154 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a95800e962c583503e5aa7bd4c2d76e57a10d386554236ab6b8eec952348c535\": container with ID starting with a95800e962c583503e5aa7bd4c2d76e57a10d386554236ab6b8eec952348c535 not found: ID does not exist" containerID="a95800e962c583503e5aa7bd4c2d76e57a10d386554236ab6b8eec952348c535" Nov 03 00:14:36.412244 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.412178 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a95800e962c583503e5aa7bd4c2d76e57a10d386554236ab6b8eec952348c535"} err="failed to get container status \"a95800e962c583503e5aa7bd4c2d76e57a10d386554236ab6b8eec952348c535\": rpc error: code = NotFound desc = could not find container \"a95800e962c583503e5aa7bd4c2d76e57a10d386554236ab6b8eec952348c535\": container with ID starting with a95800e962c583503e5aa7bd4c2d76e57a10d386554236ab6b8eec952348c535 not found: ID does not exist" Nov 03 00:14:36.412244 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.412195 5076 scope.go:122] "RemoveContainer" containerID="de90b89e5e6cd36d6e364b767496267f914aeb7089b9aa8ce383c4453f1a5d07" Nov 03 00:14:36.422514 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.422492 5076 scope.go:122] "RemoveContainer" containerID="de90b89e5e6cd36d6e364b767496267f914aeb7089b9aa8ce383c4453f1a5d07" Nov 03 00:14:36.422876 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:14:36.422843 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de90b89e5e6cd36d6e364b767496267f914aeb7089b9aa8ce383c4453f1a5d07\": container with ID starting with de90b89e5e6cd36d6e364b767496267f914aeb7089b9aa8ce383c4453f1a5d07 not found: ID does not exist" containerID="de90b89e5e6cd36d6e364b767496267f914aeb7089b9aa8ce383c4453f1a5d07" Nov 03 00:14:36.423034 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.422884 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de90b89e5e6cd36d6e364b767496267f914aeb7089b9aa8ce383c4453f1a5d07"} err="failed to get container status \"de90b89e5e6cd36d6e364b767496267f914aeb7089b9aa8ce383c4453f1a5d07\": rpc error: code = NotFound desc = could not find container \"de90b89e5e6cd36d6e364b767496267f914aeb7089b9aa8ce383c4453f1a5d07\": container with ID starting with de90b89e5e6cd36d6e364b767496267f914aeb7089b9aa8ce383c4453f1a5d07 not found: ID does not exist" Nov 03 00:14:36.489047 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.489010 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b45216a5-f1ba-4744-bd22-c59437513e86-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b45216a5-f1ba-4744-bd22-c59437513e86-istio-envoy\") pod \"b45216a5-f1ba-4744-bd22-c59437513e86\" (UID: \"b45216a5-f1ba-4744-bd22-c59437513e86\") " Nov 03 00:14:36.489224 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.489062 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c9698d7-4d4b-44e6-8165-106b31bcae2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c9698d7-4d4b-44e6-8165-106b31bcae2b-istio-podinfo\") pod \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\" (UID: \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\") " Nov 03 00:14:36.489224 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.489101 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b45216a5-f1ba-4744-bd22-c59437513e86-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b45216a5-f1ba-4744-bd22-c59437513e86-workload-certs\") pod \"b45216a5-f1ba-4744-bd22-c59437513e86\" (UID: \"b45216a5-f1ba-4744-bd22-c59437513e86\") " Nov 03 00:14:36.489224 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.489122 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c9698d7-4d4b-44e6-8165-106b31bcae2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c9698d7-4d4b-44e6-8165-106b31bcae2b-istiod-ca-cert\") pod \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\" (UID: \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\") " Nov 03 00:14:36.489224 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.489179 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b45216a5-f1ba-4744-bd22-c59437513e86-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45216a5-f1ba-4744-bd22-c59437513e86-credential-socket\") pod \"b45216a5-f1ba-4744-bd22-c59437513e86\" (UID: \"b45216a5-f1ba-4744-bd22-c59437513e86\") " Nov 03 00:14:36.489224 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.489216 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b45216a5-f1ba-4744-bd22-c59437513e86-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b45216a5-f1ba-4744-bd22-c59437513e86-istio-data\") pod \"b45216a5-f1ba-4744-bd22-c59437513e86\" (UID: \"b45216a5-f1ba-4744-bd22-c59437513e86\") " Nov 03 00:14:36.489547 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.489242 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b45216a5-f1ba-4744-bd22-c59437513e86-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45216a5-f1ba-4744-bd22-c59437513e86-workload-socket\") pod \"b45216a5-f1ba-4744-bd22-c59437513e86\" (UID: \"b45216a5-f1ba-4744-bd22-c59437513e86\") " Nov 03 00:14:36.489547 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.489323 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c9698d7-4d4b-44e6-8165-106b31bcae2b-istio-token\" (UniqueName: \"kubernetes.io/projected/5c9698d7-4d4b-44e6-8165-106b31bcae2b-istio-token\") pod \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\" (UID: \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\") " Nov 03 00:14:36.489547 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.489354 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c9698d7-4d4b-44e6-8165-106b31bcae2b-kube-api-access-llmjc\" (UniqueName: \"kubernetes.io/projected/5c9698d7-4d4b-44e6-8165-106b31bcae2b-kube-api-access-llmjc\") pod \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\" (UID: \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\") " Nov 03 00:14:36.489547 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.489427 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45216a5-f1ba-4744-bd22-c59437513e86-workload-certs" pod "b45216a5-f1ba-4744-bd22-c59437513e86" (UID: "b45216a5-f1ba-4744-bd22-c59437513e86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:36.489547 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.489435 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c9698d7-4d4b-44e6-8165-106b31bcae2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9698d7-4d4b-44e6-8165-106b31bcae2b-credential-socket\") pod \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\" (UID: \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\") " Nov 03 00:14:36.489547 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.489536 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c9698d7-4d4b-44e6-8165-106b31bcae2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9698d7-4d4b-44e6-8165-106b31bcae2b-workload-socket\") pod \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\" (UID: \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\") " Nov 03 00:14:36.489875 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.489545 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c9698d7-4d4b-44e6-8165-106b31bcae2b-istiod-ca-cert" pod "5c9698d7-4d4b-44e6-8165-106b31bcae2b" (UID: "5c9698d7-4d4b-44e6-8165-106b31bcae2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:14:36.489875 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.489639 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c9698d7-4d4b-44e6-8165-106b31bcae2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c9698d7-4d4b-44e6-8165-106b31bcae2b-workload-certs\") pod \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\" (UID: \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\") " Nov 03 00:14:36.489875 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.489656 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c9698d7-4d4b-44e6-8165-106b31bcae2b-credential-socket" pod "5c9698d7-4d4b-44e6-8165-106b31bcae2b" (UID: "5c9698d7-4d4b-44e6-8165-106b31bcae2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:36.489875 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.489666 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c9698d7-4d4b-44e6-8165-106b31bcae2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c9698d7-4d4b-44e6-8165-106b31bcae2b-istio-data\") pod \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\" (UID: \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\") " Nov 03 00:14:36.489875 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.489667 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45216a5-f1ba-4744-bd22-c59437513e86-workload-socket" pod "b45216a5-f1ba-4744-bd22-c59437513e86" (UID: "b45216a5-f1ba-4744-bd22-c59437513e86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:36.489875 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.489701 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b45216a5-f1ba-4744-bd22-c59437513e86-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b45216a5-f1ba-4744-bd22-c59437513e86-istiod-ca-cert\") pod \"b45216a5-f1ba-4744-bd22-c59437513e86\" (UID: \"b45216a5-f1ba-4744-bd22-c59437513e86\") " Nov 03 00:14:36.489875 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.489732 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b45216a5-f1ba-4744-bd22-c59437513e86-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b45216a5-f1ba-4744-bd22-c59437513e86-istio-podinfo\") pod \"b45216a5-f1ba-4744-bd22-c59437513e86\" (UID: \"b45216a5-f1ba-4744-bd22-c59437513e86\") " Nov 03 00:14:36.489875 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.489762 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c9698d7-4d4b-44e6-8165-106b31bcae2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c9698d7-4d4b-44e6-8165-106b31bcae2b-istio-envoy\") pod \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\" (UID: \"5c9698d7-4d4b-44e6-8165-106b31bcae2b\") " Nov 03 00:14:36.489875 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.489813 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b45216a5-f1ba-4744-bd22-c59437513e86-istio-token\" (UniqueName: \"kubernetes.io/projected/b45216a5-f1ba-4744-bd22-c59437513e86-istio-token\") pod \"b45216a5-f1ba-4744-bd22-c59437513e86\" (UID: \"b45216a5-f1ba-4744-bd22-c59437513e86\") " Nov 03 00:14:36.489875 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.489860 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b45216a5-f1ba-4744-bd22-c59437513e86-kube-api-access-j86gn\" (UniqueName: \"kubernetes.io/projected/b45216a5-f1ba-4744-bd22-c59437513e86-kube-api-access-j86gn\") pod \"b45216a5-f1ba-4744-bd22-c59437513e86\" (UID: \"b45216a5-f1ba-4744-bd22-c59437513e86\") " Nov 03 00:14:36.493965 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.489878 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45216a5-f1ba-4744-bd22-c59437513e86-credential-socket" pod "b45216a5-f1ba-4744-bd22-c59437513e86" (UID: "b45216a5-f1ba-4744-bd22-c59437513e86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:36.493965 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.490210 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b45216a5-f1ba-4744-bd22-c59437513e86-istiod-ca-cert" pod "b45216a5-f1ba-4744-bd22-c59437513e86" (UID: "b45216a5-f1ba-4744-bd22-c59437513e86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:14:36.493965 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.490211 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c9698d7-4d4b-44e6-8165-106b31bcae2b-workload-certs" pod "5c9698d7-4d4b-44e6-8165-106b31bcae2b" (UID: "5c9698d7-4d4b-44e6-8165-106b31bcae2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:36.493965 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.490412 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c9698d7-4d4b-44e6-8165-106b31bcae2b-workload-socket" pod "5c9698d7-4d4b-44e6-8165-106b31bcae2b" (UID: "5c9698d7-4d4b-44e6-8165-106b31bcae2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:36.493965 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.490618 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c9698d7-4d4b-44e6-8165-106b31bcae2b-istio-data" pod "5c9698d7-4d4b-44e6-8165-106b31bcae2b" (UID: "5c9698d7-4d4b-44e6-8165-106b31bcae2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:36.493965 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.490882 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45216a5-f1ba-4744-bd22-c59437513e86-istio-data" pod "b45216a5-f1ba-4744-bd22-c59437513e86" (UID: "b45216a5-f1ba-4744-bd22-c59437513e86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:36.493965 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.491020 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/905534e2-1b25-484c-9030-0185f3148cec-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:36.493965 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.491041 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b45216a5-f1ba-4744-bd22-c59437513e86-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:36.493965 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.491059 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c9698d7-4d4b-44e6-8165-106b31bcae2b-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:36.493965 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.491076 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45216a5-f1ba-4744-bd22-c59437513e86-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:36.493965 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.491090 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b45216a5-f1ba-4744-bd22-c59437513e86-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:36.493965 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.491106 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45216a5-f1ba-4744-bd22-c59437513e86-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:36.493965 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.491119 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/905534e2-1b25-484c-9030-0185f3148cec-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:36.493965 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.491132 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l56l\" (UniqueName: \"kubernetes.io/projected/905534e2-1b25-484c-9030-0185f3148cec-kube-api-access-5l56l\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:36.493965 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.491152 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9698d7-4d4b-44e6-8165-106b31bcae2b-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:36.493965 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.491170 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9698d7-4d4b-44e6-8165-106b31bcae2b-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:36.493965 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.491184 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c9698d7-4d4b-44e6-8165-106b31bcae2b-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:36.493965 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.491197 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c9698d7-4d4b-44e6-8165-106b31bcae2b-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:36.493965 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.491210 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b45216a5-f1ba-4744-bd22-c59437513e86-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:36.493965 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.491224 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/905534e2-1b25-484c-9030-0185f3148cec-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:36.493965 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.491927 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c9698d7-4d4b-44e6-8165-106b31bcae2b-istio-podinfo" pod "5c9698d7-4d4b-44e6-8165-106b31bcae2b" (UID: "5c9698d7-4d4b-44e6-8165-106b31bcae2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:14:36.493965 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.492096 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c9698d7-4d4b-44e6-8165-106b31bcae2b-kube-api-access-llmjc" pod "5c9698d7-4d4b-44e6-8165-106b31bcae2b" (UID: "5c9698d7-4d4b-44e6-8165-106b31bcae2b"). InnerVolumeSpecName "kube-api-access-llmjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:14:36.493965 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.493025 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b45216a5-f1ba-4744-bd22-c59437513e86-kube-api-access-j86gn" pod "b45216a5-f1ba-4744-bd22-c59437513e86" (UID: "b45216a5-f1ba-4744-bd22-c59437513e86"). InnerVolumeSpecName "kube-api-access-j86gn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:14:36.495228 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.494997 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b45216a5-f1ba-4744-bd22-c59437513e86-istio-podinfo" pod "b45216a5-f1ba-4744-bd22-c59437513e86" (UID: "b45216a5-f1ba-4744-bd22-c59437513e86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:14:36.495228 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.495175 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b45216a5-f1ba-4744-bd22-c59437513e86-istio-token" pod "b45216a5-f1ba-4744-bd22-c59437513e86" (UID: "b45216a5-f1ba-4744-bd22-c59437513e86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:14:36.495346 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.495302 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c9698d7-4d4b-44e6-8165-106b31bcae2b-istio-envoy" pod "5c9698d7-4d4b-44e6-8165-106b31bcae2b" (UID: "5c9698d7-4d4b-44e6-8165-106b31bcae2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:36.495570 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.495540 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c9698d7-4d4b-44e6-8165-106b31bcae2b-istio-token" pod "5c9698d7-4d4b-44e6-8165-106b31bcae2b" (UID: "5c9698d7-4d4b-44e6-8165-106b31bcae2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:14:36.496196 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.496173 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45216a5-f1ba-4744-bd22-c59437513e86-istio-envoy" pod "b45216a5-f1ba-4744-bd22-c59437513e86" (UID: "b45216a5-f1ba-4744-bd22-c59437513e86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:36.592067 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.591994 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c9698d7-4d4b-44e6-8165-106b31bcae2b-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:36.592067 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.592025 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llmjc\" (UniqueName: \"kubernetes.io/projected/5c9698d7-4d4b-44e6-8165-106b31bcae2b-kube-api-access-llmjc\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:36.592067 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.592036 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b45216a5-f1ba-4744-bd22-c59437513e86-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:36.592067 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.592044 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c9698d7-4d4b-44e6-8165-106b31bcae2b-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:36.592067 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.592055 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b45216a5-f1ba-4744-bd22-c59437513e86-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:36.592067 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.592065 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j86gn\" (UniqueName: \"kubernetes.io/projected/b45216a5-f1ba-4744-bd22-c59437513e86-kube-api-access-j86gn\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:36.592620 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.592076 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b45216a5-f1ba-4744-bd22-c59437513e86-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:36.592620 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.592084 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c9698d7-4d4b-44e6-8165-106b31bcae2b-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:36.834990 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.834940 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx"] Nov 03 00:14:36.839081 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.839053 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-89gnx"] Nov 03 00:14:36.936385 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.936358 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p"] Nov 03 00:14:36.943173 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:36.943143 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vx88p"] Nov 03 00:14:37.046510 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:37.046476 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh"] Nov 03 00:14:37.050801 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:37.050770 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-66nkh"] Nov 03 00:14:37.148817 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:37.148773 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c9698d7-4d4b-44e6-8165-106b31bcae2b" path="/var/lib/kubelet/pods/5c9698d7-4d4b-44e6-8165-106b31bcae2b/volumes" Nov 03 00:14:37.149463 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:37.149439 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="905534e2-1b25-484c-9030-0185f3148cec" path="/var/lib/kubelet/pods/905534e2-1b25-484c-9030-0185f3148cec/volumes" Nov 03 00:14:37.149998 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:37.149978 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b45216a5-f1ba-4744-bd22-c59437513e86" path="/var/lib/kubelet/pods/b45216a5-f1ba-4744-bd22-c59437513e86/volumes" Nov 03 00:14:39.884554 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:14:39.884516 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:14:50.359255 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:50.359216 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl"] Nov 03 00:14:50.360025 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:50.359500 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" podUID="6ae791d6-2d7e-4a26-9904-0b7afb962ff8" containerName="istio-proxy" containerID="cri-o://bd45930fc808e7fa71778599884a30f1adb26e4ca92dfaaabf7967e72a0e4bed" gracePeriod=30 Nov 03 00:14:55.556657 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:55.556625 5076 generic.go:356] "Generic (PLEG): container finished" podID="6ae791d6-2d7e-4a26-9904-0b7afb962ff8" containerID="bd45930fc808e7fa71778599884a30f1adb26e4ca92dfaaabf7967e72a0e4bed" exitCode=0 Nov 03 00:14:55.557091 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:55.556706 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" event={"ID":"6ae791d6-2d7e-4a26-9904-0b7afb962ff8","Type":"ContainerDied","Data":"bd45930fc808e7fa71778599884a30f1adb26e4ca92dfaaabf7967e72a0e4bed"} Nov 03 00:14:55.557091 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:55.556785 5076 scope.go:122] "RemoveContainer" containerID="2b52e5d661a97e1ea6a7706f43b26aea9efad61bd78431d3b56fad3b5ea399e7" Nov 03 00:14:55.642049 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:55.641985 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" Nov 03 00:14:55.743421 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:55.743390 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-credential-socket\") pod \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\" (UID: \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\") " Nov 03 00:14:55.743619 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:55.743464 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-kube-api-access-vjdb9\" (UniqueName: \"kubernetes.io/projected/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-kube-api-access-vjdb9\") pod \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\" (UID: \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\") " Nov 03 00:14:55.743619 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:55.743490 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-istio-podinfo\") pod \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\" (UID: \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\") " Nov 03 00:14:55.743619 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:55.743510 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-istiod-ca-cert\") pod \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\" (UID: \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\") " Nov 03 00:14:55.743619 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:55.743555 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-workload-socket\") pod \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\" (UID: \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\") " Nov 03 00:14:55.743619 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:55.743603 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-workload-certs\") pod \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\" (UID: \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\") " Nov 03 00:14:55.743926 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:55.743629 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-istio-envoy\") pod \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\" (UID: \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\") " Nov 03 00:14:55.743926 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:55.743700 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-istio-data\") pod \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\" (UID: \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\") " Nov 03 00:14:55.743926 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:55.743699 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-credential-socket" pod "6ae791d6-2d7e-4a26-9904-0b7afb962ff8" (UID: "6ae791d6-2d7e-4a26-9904-0b7afb962ff8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:55.743926 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:55.743758 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-istio-token\" (UniqueName: \"kubernetes.io/projected/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-istio-token\") pod \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\" (UID: \"6ae791d6-2d7e-4a26-9904-0b7afb962ff8\") " Nov 03 00:14:55.744158 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:55.744018 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-workload-socket" pod "6ae791d6-2d7e-4a26-9904-0b7afb962ff8" (UID: "6ae791d6-2d7e-4a26-9904-0b7afb962ff8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:55.744289 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:55.744266 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:55.744440 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:55.744291 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:55.744504 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:55.744438 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-istio-data" pod "6ae791d6-2d7e-4a26-9904-0b7afb962ff8" (UID: "6ae791d6-2d7e-4a26-9904-0b7afb962ff8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:55.744573 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:55.744499 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-istiod-ca-cert" pod "6ae791d6-2d7e-4a26-9904-0b7afb962ff8" (UID: "6ae791d6-2d7e-4a26-9904-0b7afb962ff8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:14:55.744635 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:55.744612 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-workload-certs" pod "6ae791d6-2d7e-4a26-9904-0b7afb962ff8" (UID: "6ae791d6-2d7e-4a26-9904-0b7afb962ff8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:55.746203 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:55.746175 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-istio-podinfo" pod "6ae791d6-2d7e-4a26-9904-0b7afb962ff8" (UID: "6ae791d6-2d7e-4a26-9904-0b7afb962ff8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:14:55.746324 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:55.746178 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-istio-envoy" pod "6ae791d6-2d7e-4a26-9904-0b7afb962ff8" (UID: "6ae791d6-2d7e-4a26-9904-0b7afb962ff8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:55.746324 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:55.746223 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-kube-api-access-vjdb9" pod "6ae791d6-2d7e-4a26-9904-0b7afb962ff8" (UID: "6ae791d6-2d7e-4a26-9904-0b7afb962ff8"). InnerVolumeSpecName "kube-api-access-vjdb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:14:55.746583 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:55.746565 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-istio-token" pod "6ae791d6-2d7e-4a26-9904-0b7afb962ff8" (UID: "6ae791d6-2d7e-4a26-9904-0b7afb962ff8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:14:55.845184 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:55.845150 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjdb9\" (UniqueName: \"kubernetes.io/projected/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-kube-api-access-vjdb9\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:55.845184 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:55.845180 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:55.845184 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:55.845190 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:55.845541 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:55.845199 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:55.845541 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:55.845208 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:55.845541 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:55.845216 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:55.845541 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:55.845224 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ae791d6-2d7e-4a26-9904-0b7afb962ff8-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:56.565013 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:56.564981 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" Nov 03 00:14:56.565527 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:56.565018 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl" event={"ID":"6ae791d6-2d7e-4a26-9904-0b7afb962ff8","Type":"ContainerDied","Data":"449e1e7efbc10e351f6d3252b0df706be0c3c32812f40942c82eaf3b2b61d5cd"} Nov 03 00:14:56.565527 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:56.565085 5076 scope.go:122] "RemoveContainer" containerID="bd45930fc808e7fa71778599884a30f1adb26e4ca92dfaaabf7967e72a0e4bed" Nov 03 00:14:56.769966 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:56.769927 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl"] Nov 03 00:14:56.817362 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:56.817276 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1f668443-openshift-default-5cd46c77b9-kx9wl"] Nov 03 00:14:57.181998 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:14:57.181713 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ae791d6-2d7e-4a26-9904-0b7afb962ff8" path="/var/lib/kubelet/pods/6ae791d6-2d7e-4a26-9904-0b7afb962ff8/volumes" Nov 03 00:15:00.182513 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.182478 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894415-h9zww"] Nov 03 00:15:00.183070 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.183049 5076 cpu_manager.go:414] "containerMap: removing container" podUID="6ae791d6-2d7e-4a26-9904-0b7afb962ff8" containerName="istio-proxy" Nov 03 00:15:00.183070 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.183064 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ae791d6-2d7e-4a26-9904-0b7afb962ff8" containerName="istio-proxy" Nov 03 00:15:00.183186 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.183077 5076 cpu_manager.go:414] "containerMap: removing container" podUID="905534e2-1b25-484c-9030-0185f3148cec" containerName="istio-proxy" Nov 03 00:15:00.183186 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.183082 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="905534e2-1b25-484c-9030-0185f3148cec" containerName="istio-proxy" Nov 03 00:15:00.183186 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.183089 5076 cpu_manager.go:414] "containerMap: removing container" podUID="0d78fb30-3ea3-45c8-9761-31fe6b941839" containerName="istio-proxy" Nov 03 00:15:00.183186 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.183094 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d78fb30-3ea3-45c8-9761-31fe6b941839" containerName="istio-proxy" Nov 03 00:15:00.183186 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.183107 5076 cpu_manager.go:414] "containerMap: removing container" podUID="5c9698d7-4d4b-44e6-8165-106b31bcae2b" containerName="istio-proxy" Nov 03 00:15:00.183186 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.183116 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c9698d7-4d4b-44e6-8165-106b31bcae2b" containerName="istio-proxy" Nov 03 00:15:00.183186 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.183134 5076 cpu_manager.go:414] "containerMap: removing container" podUID="6ae791d6-2d7e-4a26-9904-0b7afb962ff8" containerName="istio-proxy" Nov 03 00:15:00.183186 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.183139 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ae791d6-2d7e-4a26-9904-0b7afb962ff8" containerName="istio-proxy" Nov 03 00:15:00.183186 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.183158 5076 cpu_manager.go:414] "containerMap: removing container" podUID="b45216a5-f1ba-4744-bd22-c59437513e86" containerName="istio-proxy" Nov 03 00:15:00.183186 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.183163 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b45216a5-f1ba-4744-bd22-c59437513e86" containerName="istio-proxy" Nov 03 00:15:00.183715 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.183240 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="b45216a5-f1ba-4744-bd22-c59437513e86" containerName="istio-proxy" Nov 03 00:15:00.183715 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.183253 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ae791d6-2d7e-4a26-9904-0b7afb962ff8" containerName="istio-proxy" Nov 03 00:15:00.183715 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.183259 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="905534e2-1b25-484c-9030-0185f3148cec" containerName="istio-proxy" Nov 03 00:15:00.183715 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.183265 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d78fb30-3ea3-45c8-9761-31fe6b941839" containerName="istio-proxy" Nov 03 00:15:00.183715 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.183272 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c9698d7-4d4b-44e6-8165-106b31bcae2b" containerName="istio-proxy" Nov 03 00:15:00.183715 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.183280 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ae791d6-2d7e-4a26-9904-0b7afb962ff8" containerName="istio-proxy" Nov 03 00:15:00.187852 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.187830 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894415-5gmm7"] Nov 03 00:15:00.188018 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.187975 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894415-h9zww" Nov 03 00:15:00.191672 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.191651 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894415-5gmm7" Nov 03 00:15:00.237220 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.237195 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 03 00:15:00.296097 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.296064 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894415-h9zww"] Nov 03 00:15:00.306983 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.306951 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894415-5gmm7"] Nov 03 00:15:00.321708 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.321673 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d4868a3-1492-4d61-81ae-080156be2b73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ba1918-29894415-5gmm7\" (UID: \"4d4868a3-1492-4d61-81ae-080156be2b73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894415-5gmm7" Nov 03 00:15:00.321940 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.321758 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmfht\" (UniqueName: \"kubernetes.io/projected/4d4868a3-1492-4d61-81ae-080156be2b73-kube-api-access-jmfht\") pod \"maas-api-key-cleanup-e2e-worker-w2-ba1918-29894415-5gmm7\" (UID: \"4d4868a3-1492-4d61-81ae-080156be2b73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894415-5gmm7" Nov 03 00:15:00.322042 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.321983 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh6m4\" (UniqueName: \"kubernetes.io/projected/9e58f4ff-b1f9-45d4-95ad-e628431da7b7-kube-api-access-bh6m4\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894415-h9zww\" (UID: \"9e58f4ff-b1f9-45d4-95ad-e628431da7b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894415-h9zww" Nov 03 00:15:00.322114 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.322067 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e58f4ff-b1f9-45d4-95ad-e628431da7b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894415-h9zww\" (UID: \"9e58f4ff-b1f9-45d4-95ad-e628431da7b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894415-h9zww" Nov 03 00:15:00.423427 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.423392 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmfht\" (UniqueName: \"kubernetes.io/projected/4d4868a3-1492-4d61-81ae-080156be2b73-kube-api-access-jmfht\") pod \"maas-api-key-cleanup-e2e-worker-w2-ba1918-29894415-5gmm7\" (UID: \"4d4868a3-1492-4d61-81ae-080156be2b73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894415-5gmm7" Nov 03 00:15:00.423623 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.423491 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh6m4\" (UniqueName: \"kubernetes.io/projected/9e58f4ff-b1f9-45d4-95ad-e628431da7b7-kube-api-access-bh6m4\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894415-h9zww\" (UID: \"9e58f4ff-b1f9-45d4-95ad-e628431da7b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894415-h9zww" Nov 03 00:15:00.423623 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.423526 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e58f4ff-b1f9-45d4-95ad-e628431da7b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894415-h9zww\" (UID: \"9e58f4ff-b1f9-45d4-95ad-e628431da7b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894415-h9zww" Nov 03 00:15:00.423623 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.423583 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d4868a3-1492-4d61-81ae-080156be2b73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ba1918-29894415-5gmm7\" (UID: \"4d4868a3-1492-4d61-81ae-080156be2b73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894415-5gmm7" Nov 03 00:15:00.424441 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.424418 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d4868a3-1492-4d61-81ae-080156be2b73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ba1918-29894415-5gmm7\" (UID: \"4d4868a3-1492-4d61-81ae-080156be2b73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894415-5gmm7" Nov 03 00:15:00.424499 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.424418 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e58f4ff-b1f9-45d4-95ad-e628431da7b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894415-h9zww\" (UID: \"9e58f4ff-b1f9-45d4-95ad-e628431da7b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894415-h9zww" Nov 03 00:15:00.476972 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.476942 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh6m4\" (UniqueName: \"kubernetes.io/projected/9e58f4ff-b1f9-45d4-95ad-e628431da7b7-kube-api-access-bh6m4\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894415-h9zww\" (UID: \"9e58f4ff-b1f9-45d4-95ad-e628431da7b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894415-h9zww" Nov 03 00:15:00.477136 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.477040 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmfht\" (UniqueName: \"kubernetes.io/projected/4d4868a3-1492-4d61-81ae-080156be2b73-kube-api-access-jmfht\") pod \"maas-api-key-cleanup-e2e-worker-w2-ba1918-29894415-5gmm7\" (UID: \"4d4868a3-1492-4d61-81ae-080156be2b73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894415-5gmm7" Nov 03 00:15:00.505078 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.505046 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894415-h9zww" Nov 03 00:15:00.510428 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.510407 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894415-5gmm7" Nov 03 00:15:00.880455 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.880425 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894415-h9zww"] Nov 03 00:15:00.881028 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:15:00.881003 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e58f4ff_b1f9_45d4_95ad_e628431da7b7.slice/crio-1225306d740d73a9accdcb70f47e397531eb60855ccca980818acaf7ca52ffeb WatchSource:0}: Error finding container 1225306d740d73a9accdcb70f47e397531eb60855ccca980818acaf7ca52ffeb: Status 404 returned error can't find the container with id 1225306d740d73a9accdcb70f47e397531eb60855ccca980818acaf7ca52ffeb Nov 03 00:15:00.926142 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:00.926109 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894415-5gmm7"] Nov 03 00:15:00.927459 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:15:00.927425 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d4868a3_1492_4d61_81ae_080156be2b73.slice/crio-ed5349bd7d39d5c6356236247f9e6ed4778efb14f7cad22cfcd5ae4cb8228460 WatchSource:0}: Error finding container ed5349bd7d39d5c6356236247f9e6ed4778efb14f7cad22cfcd5ae4cb8228460: Status 404 returned error can't find the container with id ed5349bd7d39d5c6356236247f9e6ed4778efb14f7cad22cfcd5ae4cb8228460 Nov 03 00:15:01.604641 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:01.604551 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894415-5gmm7" event={"ID":"4d4868a3-1492-4d61-81ae-080156be2b73","Type":"ContainerStarted","Data":"36c5aabc75fa5927a529194f7aa8af0cd1ab85009cc85ddf1eb0f54bf0eb4018"} Nov 03 00:15:01.604641 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:01.604613 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894415-5gmm7" event={"ID":"4d4868a3-1492-4d61-81ae-080156be2b73","Type":"ContainerStarted","Data":"ed5349bd7d39d5c6356236247f9e6ed4778efb14f7cad22cfcd5ae4cb8228460"} Nov 03 00:15:01.606261 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:01.606231 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894415-h9zww" event={"ID":"9e58f4ff-b1f9-45d4-95ad-e628431da7b7","Type":"ContainerStarted","Data":"bcd09271907e767fa2f981358cd54c739218154cec0dd1f140dedf58dafa40a1"} Nov 03 00:15:01.606368 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:01.606289 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894415-h9zww" event={"ID":"9e58f4ff-b1f9-45d4-95ad-e628431da7b7","Type":"ContainerStarted","Data":"1225306d740d73a9accdcb70f47e397531eb60855ccca980818acaf7ca52ffeb"} Nov 03 00:15:01.713547 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:01.713489 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894415-5gmm7" podStartSLOduration=1.713473156 podStartE2EDuration="1.713473156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:01.711971659 +0000 UTC m=+4950.680503443" watchObservedRunningTime="2026-11-03 00:15:01.713473156 +0000 UTC m=+4950.682004951" Nov 03 00:15:01.818780 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:01.818677 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894415-h9zww" podStartSLOduration=1.818663925 podStartE2EDuration="1.818663925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:01.817589992 +0000 UTC m=+4950.786121777" watchObservedRunningTime="2026-11-03 00:15:01.818663925 +0000 UTC m=+4950.787195708" Nov 03 00:15:02.614148 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:02.614111 5076 generic.go:356] "Generic (PLEG): container finished" podID="9e58f4ff-b1f9-45d4-95ad-e628431da7b7" containerID="bcd09271907e767fa2f981358cd54c739218154cec0dd1f140dedf58dafa40a1" exitCode=60 Nov 03 00:15:02.614570 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:02.614198 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894415-h9zww" event={"ID":"9e58f4ff-b1f9-45d4-95ad-e628431da7b7","Type":"ContainerDied","Data":"bcd09271907e767fa2f981358cd54c739218154cec0dd1f140dedf58dafa40a1"} Nov 03 00:15:02.614570 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:02.614478 5076 scope.go:122] "RemoveContainer" containerID="bcd09271907e767fa2f981358cd54c739218154cec0dd1f140dedf58dafa40a1" Nov 03 00:15:02.615938 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:02.615913 5076 generic.go:356] "Generic (PLEG): container finished" podID="4d4868a3-1492-4d61-81ae-080156be2b73" containerID="36c5aabc75fa5927a529194f7aa8af0cd1ab85009cc85ddf1eb0f54bf0eb4018" exitCode=60 Nov 03 00:15:02.616074 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:02.615960 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894415-5gmm7" event={"ID":"4d4868a3-1492-4d61-81ae-080156be2b73","Type":"ContainerDied","Data":"36c5aabc75fa5927a529194f7aa8af0cd1ab85009cc85ddf1eb0f54bf0eb4018"} Nov 03 00:15:02.616262 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:02.616245 5076 scope.go:122] "RemoveContainer" containerID="36c5aabc75fa5927a529194f7aa8af0cd1ab85009cc85ddf1eb0f54bf0eb4018" Nov 03 00:15:03.624387 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:03.624352 5076 generic.go:356] "Generic (PLEG): container finished" podID="9e58f4ff-b1f9-45d4-95ad-e628431da7b7" containerID="928346256f41559d620292003f3093626ca96451543b29bac8fcb4661b3610c1" exitCode=60 Nov 03 00:15:03.624823 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:03.624431 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894415-h9zww" event={"ID":"9e58f4ff-b1f9-45d4-95ad-e628431da7b7","Type":"ContainerDied","Data":"928346256f41559d620292003f3093626ca96451543b29bac8fcb4661b3610c1"} Nov 03 00:15:03.624823 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:03.624495 5076 scope.go:122] "RemoveContainer" containerID="bcd09271907e767fa2f981358cd54c739218154cec0dd1f140dedf58dafa40a1" Nov 03 00:15:03.624823 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:03.624765 5076 scope.go:122] "RemoveContainer" containerID="928346256f41559d620292003f3093626ca96451543b29bac8fcb4661b3610c1" Nov 03 00:15:03.625110 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:15:03.625086 5076 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894415-h9zww_odh-ai-gateway-infra(9e58f4ff-b1f9-45d4-95ad-e628431da7b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894415-h9zww" podUID="9e58f4ff-b1f9-45d4-95ad-e628431da7b7" Nov 03 00:15:03.626709 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:03.626687 5076 generic.go:356] "Generic (PLEG): container finished" podID="4d4868a3-1492-4d61-81ae-080156be2b73" containerID="7f568c77452b087e7f15bba949ed76185b48c933974cca336bbac716390fb55b" exitCode=60 Nov 03 00:15:03.626849 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:03.626748 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894415-5gmm7" event={"ID":"4d4868a3-1492-4d61-81ae-080156be2b73","Type":"ContainerDied","Data":"7f568c77452b087e7f15bba949ed76185b48c933974cca336bbac716390fb55b"} Nov 03 00:15:03.627051 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:03.627034 5076 scope.go:122] "RemoveContainer" containerID="7f568c77452b087e7f15bba949ed76185b48c933974cca336bbac716390fb55b" Nov 03 00:15:03.627335 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:15:03.627316 5076 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ba1918-29894415-5gmm7_odh-ai-gateway-infra(4d4868a3-1492-4d61-81ae-080156be2b73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894415-5gmm7" podUID="4d4868a3-1492-4d61-81ae-080156be2b73" Nov 03 00:15:03.636744 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:03.636716 5076 scope.go:122] "RemoveContainer" containerID="36c5aabc75fa5927a529194f7aa8af0cd1ab85009cc85ddf1eb0f54bf0eb4018" Nov 03 00:15:04.633873 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:04.633839 5076 scope.go:122] "RemoveContainer" containerID="7f568c77452b087e7f15bba949ed76185b48c933974cca336bbac716390fb55b" Nov 03 00:15:04.634368 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:15:04.634174 5076 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ba1918-29894415-5gmm7_odh-ai-gateway-infra(4d4868a3-1492-4d61-81ae-080156be2b73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894415-5gmm7" podUID="4d4868a3-1492-4d61-81ae-080156be2b73" Nov 03 00:15:04.635329 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:04.635309 5076 scope.go:122] "RemoveContainer" containerID="928346256f41559d620292003f3093626ca96451543b29bac8fcb4661b3610c1" Nov 03 00:15:04.635562 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:15:04.635537 5076 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894415-h9zww_odh-ai-gateway-infra(9e58f4ff-b1f9-45d4-95ad-e628431da7b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894415-h9zww" podUID="9e58f4ff-b1f9-45d4-95ad-e628431da7b7" Nov 03 00:15:09.884951 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:15:09.884882 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:15:11.561111 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:11.561072 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-fe63ac/rate-test-model-fe63ac-kserve-54cb8c857c-lntw7"] Nov 03 00:15:15.141403 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:15.141368 5076 scope.go:122] "RemoveContainer" containerID="928346256f41559d620292003f3093626ca96451543b29bac8fcb4661b3610c1" Nov 03 00:15:15.703982 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:15.703946 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894415-h9zww" event={"ID":"9e58f4ff-b1f9-45d4-95ad-e628431da7b7","Type":"ContainerStarted","Data":"375d745dc023434fcdc458eb47c670467bfc417844142614ac18154bd13f5e67"} Nov 03 00:15:16.712028 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:16.711989 5076 generic.go:356] "Generic (PLEG): container finished" podID="9e58f4ff-b1f9-45d4-95ad-e628431da7b7" containerID="375d745dc023434fcdc458eb47c670467bfc417844142614ac18154bd13f5e67" exitCode=60 Nov 03 00:15:16.712517 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:16.712057 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894415-h9zww" event={"ID":"9e58f4ff-b1f9-45d4-95ad-e628431da7b7","Type":"ContainerDied","Data":"375d745dc023434fcdc458eb47c670467bfc417844142614ac18154bd13f5e67"} Nov 03 00:15:16.712517 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:16.712133 5076 scope.go:122] "RemoveContainer" containerID="928346256f41559d620292003f3093626ca96451543b29bac8fcb4661b3610c1" Nov 03 00:15:16.712517 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:16.712501 5076 scope.go:122] "RemoveContainer" containerID="375d745dc023434fcdc458eb47c670467bfc417844142614ac18154bd13f5e67" Nov 03 00:15:16.712793 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:15:16.712769 5076 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894415-h9zww_odh-ai-gateway-infra(9e58f4ff-b1f9-45d4-95ad-e628431da7b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894415-h9zww" podUID="9e58f4ff-b1f9-45d4-95ad-e628431da7b7" Nov 03 00:15:16.822018 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:16.821982 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894415-h9zww"] Nov 03 00:15:17.141045 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:17.140964 5076 scope.go:122] "RemoveContainer" containerID="7f568c77452b087e7f15bba949ed76185b48c933974cca336bbac716390fb55b" Nov 03 00:15:17.720484 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:17.720445 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894415-5gmm7" event={"ID":"4d4868a3-1492-4d61-81ae-080156be2b73","Type":"ContainerStarted","Data":"dcad49e186cdbd42e0873d70808295ed694ab62d51bb7bb214ca87bd8eeaf17b"} Nov 03 00:15:17.898193 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:17.898158 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894415-h9zww" Nov 03 00:15:18.041410 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:18.041285 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e58f4ff-b1f9-45d4-95ad-e628431da7b7-kube-api-access-bh6m4\" (UniqueName: \"kubernetes.io/projected/9e58f4ff-b1f9-45d4-95ad-e628431da7b7-kube-api-access-bh6m4\") pod \"9e58f4ff-b1f9-45d4-95ad-e628431da7b7\" (UID: \"9e58f4ff-b1f9-45d4-95ad-e628431da7b7\") " Nov 03 00:15:18.041661 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:18.041461 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e58f4ff-b1f9-45d4-95ad-e628431da7b7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e58f4ff-b1f9-45d4-95ad-e628431da7b7-maas-api-service-ca\") pod \"9e58f4ff-b1f9-45d4-95ad-e628431da7b7\" (UID: \"9e58f4ff-b1f9-45d4-95ad-e628431da7b7\") " Nov 03 00:15:18.042786 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:18.042751 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e58f4ff-b1f9-45d4-95ad-e628431da7b7-maas-api-service-ca" pod "9e58f4ff-b1f9-45d4-95ad-e628431da7b7" (UID: "9e58f4ff-b1f9-45d4-95ad-e628431da7b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:18.045095 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:18.045067 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e58f4ff-b1f9-45d4-95ad-e628431da7b7-kube-api-access-bh6m4" pod "9e58f4ff-b1f9-45d4-95ad-e628431da7b7" (UID: "9e58f4ff-b1f9-45d4-95ad-e628431da7b7"). InnerVolumeSpecName "kube-api-access-bh6m4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:18.142668 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:18.142636 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh6m4\" (UniqueName: \"kubernetes.io/projected/9e58f4ff-b1f9-45d4-95ad-e628431da7b7-kube-api-access-bh6m4\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:18.142668 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:18.142666 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e58f4ff-b1f9-45d4-95ad-e628431da7b7-maas-api-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:18.729525 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:18.729494 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894415-h9zww" Nov 03 00:15:18.730021 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:18.729535 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894415-h9zww" event={"ID":"9e58f4ff-b1f9-45d4-95ad-e628431da7b7","Type":"ContainerDied","Data":"1225306d740d73a9accdcb70f47e397531eb60855ccca980818acaf7ca52ffeb"} Nov 03 00:15:18.730021 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:18.729595 5076 scope.go:122] "RemoveContainer" containerID="375d745dc023434fcdc458eb47c670467bfc417844142614ac18154bd13f5e67" Nov 03 00:15:18.731713 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:18.731689 5076 generic.go:356] "Generic (PLEG): container finished" podID="4d4868a3-1492-4d61-81ae-080156be2b73" containerID="dcad49e186cdbd42e0873d70808295ed694ab62d51bb7bb214ca87bd8eeaf17b" exitCode=60 Nov 03 00:15:18.731813 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:18.731768 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894415-5gmm7" event={"ID":"4d4868a3-1492-4d61-81ae-080156be2b73","Type":"ContainerDied","Data":"dcad49e186cdbd42e0873d70808295ed694ab62d51bb7bb214ca87bd8eeaf17b"} Nov 03 00:15:18.732231 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:18.732214 5076 scope.go:122] "RemoveContainer" containerID="dcad49e186cdbd42e0873d70808295ed694ab62d51bb7bb214ca87bd8eeaf17b" Nov 03 00:15:18.732633 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:15:18.732592 5076 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ba1918-29894415-5gmm7_odh-ai-gateway-infra(4d4868a3-1492-4d61-81ae-080156be2b73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894415-5gmm7" podUID="4d4868a3-1492-4d61-81ae-080156be2b73" Nov 03 00:15:18.739631 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:18.739614 5076 scope.go:122] "RemoveContainer" containerID="7f568c77452b087e7f15bba949ed76185b48c933974cca336bbac716390fb55b" Nov 03 00:15:18.839806 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:18.839776 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894415-5gmm7"] Nov 03 00:15:18.885692 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:18.885663 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894415-h9zww"] Nov 03 00:15:18.891002 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:18.890977 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894415-h9zww"] Nov 03 00:15:19.149510 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:19.149425 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e58f4ff-b1f9-45d4-95ad-e628431da7b7" path="/var/lib/kubelet/pods/9e58f4ff-b1f9-45d4-95ad-e628431da7b7/volumes" Nov 03 00:15:19.896250 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:19.896220 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894415-5gmm7" Nov 03 00:15:19.969011 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:19.968978 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d4868a3-1492-4d61-81ae-080156be2b73-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d4868a3-1492-4d61-81ae-080156be2b73-maas-api-service-ca\") pod \"4d4868a3-1492-4d61-81ae-080156be2b73\" (UID: \"4d4868a3-1492-4d61-81ae-080156be2b73\") " Nov 03 00:15:19.969168 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:19.969101 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d4868a3-1492-4d61-81ae-080156be2b73-kube-api-access-jmfht\" (UniqueName: \"kubernetes.io/projected/4d4868a3-1492-4d61-81ae-080156be2b73-kube-api-access-jmfht\") pod \"4d4868a3-1492-4d61-81ae-080156be2b73\" (UID: \"4d4868a3-1492-4d61-81ae-080156be2b73\") " Nov 03 00:15:19.969415 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:19.969393 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d4868a3-1492-4d61-81ae-080156be2b73-maas-api-service-ca" pod "4d4868a3-1492-4d61-81ae-080156be2b73" (UID: "4d4868a3-1492-4d61-81ae-080156be2b73"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:19.969545 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:19.969530 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d4868a3-1492-4d61-81ae-080156be2b73-maas-api-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:19.971231 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:19.971207 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d4868a3-1492-4d61-81ae-080156be2b73-kube-api-access-jmfht" pod "4d4868a3-1492-4d61-81ae-080156be2b73" (UID: "4d4868a3-1492-4d61-81ae-080156be2b73"). InnerVolumeSpecName "kube-api-access-jmfht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:20.070565 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:20.070485 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmfht\" (UniqueName: \"kubernetes.io/projected/4d4868a3-1492-4d61-81ae-080156be2b73-kube-api-access-jmfht\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:20.749883 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:20.749849 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894415-5gmm7" Nov 03 00:15:20.750171 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:20.749915 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894415-5gmm7" event={"ID":"4d4868a3-1492-4d61-81ae-080156be2b73","Type":"ContainerDied","Data":"ed5349bd7d39d5c6356236247f9e6ed4778efb14f7cad22cfcd5ae4cb8228460"} Nov 03 00:15:20.750171 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:20.749980 5076 scope.go:122] "RemoveContainer" containerID="dcad49e186cdbd42e0873d70808295ed694ab62d51bb7bb214ca87bd8eeaf17b" Nov 03 00:15:20.908924 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:20.908877 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894415-5gmm7"] Nov 03 00:15:20.913110 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:20.913084 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894415-5gmm7"] Nov 03 00:15:21.148715 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:21.148638 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d4868a3-1492-4d61-81ae-080156be2b73" path="/var/lib/kubelet/pods/4d4868a3-1492-4d61-81ae-080156be2b73/volumes" Nov 03 00:15:26.581592 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:26.581524 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-fe63ac/rate-test-model-fe63ac-kserve-54cb8c857c-lntw7" podUID="af4882d8-765e-4167-9a4a-06a7d77f7437" containerName="main" containerID="cri-o://f8a26f5d8a0a4d08087e8d647a86b5c069877f116d1f337927e7b839ab95a29c" gracePeriod=45 Nov 03 00:15:26.742078 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:26.742037 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-fe63ac/rate-test-model-fe63ac-kserve-54cb8c857c-lntw7" podUID="af4882d8-765e-4167-9a4a-06a7d77f7437" containerName="main" probeResult="failure" output="Get \"https://10.132.0.199:8000/ready\": dial tcp 10.132.0.199:8000: connect: connection refused" Nov 03 00:15:26.806350 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:26.806308 5076 generic.go:356] "Generic (PLEG): container finished" podID="af4882d8-765e-4167-9a4a-06a7d77f7437" containerID="f8a26f5d8a0a4d08087e8d647a86b5c069877f116d1f337927e7b839ab95a29c" exitCode=0 Nov 03 00:15:26.806509 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:26.806393 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-fe63ac/rate-test-model-fe63ac-kserve-54cb8c857c-lntw7" event={"ID":"af4882d8-765e-4167-9a4a-06a7d77f7437","Type":"ContainerDied","Data":"f8a26f5d8a0a4d08087e8d647a86b5c069877f116d1f337927e7b839ab95a29c"} Nov 03 00:15:26.870838 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:26.870815 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-fe63ac/rate-test-model-fe63ac-kserve-54cb8c857c-lntw7" Nov 03 00:15:26.949550 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:26.949519 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af4882d8-765e-4167-9a4a-06a7d77f7437-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af4882d8-765e-4167-9a4a-06a7d77f7437-tmp-dir\") pod \"af4882d8-765e-4167-9a4a-06a7d77f7437\" (UID: \"af4882d8-765e-4167-9a4a-06a7d77f7437\") " Nov 03 00:15:26.949731 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:26.949620 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af4882d8-765e-4167-9a4a-06a7d77f7437-tls-certs\" (UniqueName: \"kubernetes.io/secret/af4882d8-765e-4167-9a4a-06a7d77f7437-tls-certs\") pod \"af4882d8-765e-4167-9a4a-06a7d77f7437\" (UID: \"af4882d8-765e-4167-9a4a-06a7d77f7437\") " Nov 03 00:15:26.949731 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:26.949661 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af4882d8-765e-4167-9a4a-06a7d77f7437-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af4882d8-765e-4167-9a4a-06a7d77f7437-model-cache\") pod \"af4882d8-765e-4167-9a4a-06a7d77f7437\" (UID: \"af4882d8-765e-4167-9a4a-06a7d77f7437\") " Nov 03 00:15:26.949731 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:26.949680 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af4882d8-765e-4167-9a4a-06a7d77f7437-dshm\" (UniqueName: \"kubernetes.io/empty-dir/af4882d8-765e-4167-9a4a-06a7d77f7437-dshm\") pod \"af4882d8-765e-4167-9a4a-06a7d77f7437\" (UID: \"af4882d8-765e-4167-9a4a-06a7d77f7437\") " Nov 03 00:15:26.949731 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:26.949703 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af4882d8-765e-4167-9a4a-06a7d77f7437-home\" (UniqueName: \"kubernetes.io/empty-dir/af4882d8-765e-4167-9a4a-06a7d77f7437-home\") pod \"af4882d8-765e-4167-9a4a-06a7d77f7437\" (UID: \"af4882d8-765e-4167-9a4a-06a7d77f7437\") " Nov 03 00:15:26.950283 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:26.949741 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af4882d8-765e-4167-9a4a-06a7d77f7437-kube-api-access-h2skw\" (UniqueName: \"kubernetes.io/projected/af4882d8-765e-4167-9a4a-06a7d77f7437-kube-api-access-h2skw\") pod \"af4882d8-765e-4167-9a4a-06a7d77f7437\" (UID: \"af4882d8-765e-4167-9a4a-06a7d77f7437\") " Nov 03 00:15:26.950283 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:26.949835 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af4882d8-765e-4167-9a4a-06a7d77f7437-tmp-dir" pod "af4882d8-765e-4167-9a4a-06a7d77f7437" (UID: "af4882d8-765e-4167-9a4a-06a7d77f7437"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:15:26.950283 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:26.949959 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af4882d8-765e-4167-9a4a-06a7d77f7437-model-cache" pod "af4882d8-765e-4167-9a4a-06a7d77f7437" (UID: "af4882d8-765e-4167-9a4a-06a7d77f7437"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:15:26.950283 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:26.950023 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af4882d8-765e-4167-9a4a-06a7d77f7437-home" pod "af4882d8-765e-4167-9a4a-06a7d77f7437" (UID: "af4882d8-765e-4167-9a4a-06a7d77f7437"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:15:26.950283 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:26.950229 5076 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af4882d8-765e-4167-9a4a-06a7d77f7437-tmp-dir\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:26.950283 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:26.950245 5076 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af4882d8-765e-4167-9a4a-06a7d77f7437-model-cache\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:26.950283 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:26.950256 5076 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af4882d8-765e-4167-9a4a-06a7d77f7437-home\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:26.951936 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:26.951912 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af4882d8-765e-4167-9a4a-06a7d77f7437-tls-certs" pod "af4882d8-765e-4167-9a4a-06a7d77f7437" (UID: "af4882d8-765e-4167-9a4a-06a7d77f7437"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:15:26.952374 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:26.952356 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af4882d8-765e-4167-9a4a-06a7d77f7437-dshm" pod "af4882d8-765e-4167-9a4a-06a7d77f7437" (UID: "af4882d8-765e-4167-9a4a-06a7d77f7437"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:15:26.952433 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:26.952407 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af4882d8-765e-4167-9a4a-06a7d77f7437-kube-api-access-h2skw" pod "af4882d8-765e-4167-9a4a-06a7d77f7437" (UID: "af4882d8-765e-4167-9a4a-06a7d77f7437"). InnerVolumeSpecName "kube-api-access-h2skw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:27.051294 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:27.051259 5076 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af4882d8-765e-4167-9a4a-06a7d77f7437-tls-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:27.051294 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:27.051291 5076 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af4882d8-765e-4167-9a4a-06a7d77f7437-dshm\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:27.051511 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:27.051303 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2skw\" (UniqueName: \"kubernetes.io/projected/af4882d8-765e-4167-9a4a-06a7d77f7437-kube-api-access-h2skw\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:27.813580 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:27.813540 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-fe63ac/rate-test-model-fe63ac-kserve-54cb8c857c-lntw7" event={"ID":"af4882d8-765e-4167-9a4a-06a7d77f7437","Type":"ContainerDied","Data":"8e027565521314fd95ca1ad461df5d6f5fe52edb7961be37c4d5fe9aa4fd94b3"} Nov 03 00:15:27.814047 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:27.813591 5076 scope.go:122] "RemoveContainer" containerID="f8a26f5d8a0a4d08087e8d647a86b5c069877f116d1f337927e7b839ab95a29c" Nov 03 00:15:27.814047 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:27.813551 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-fe63ac/rate-test-model-fe63ac-kserve-54cb8c857c-lntw7" Nov 03 00:15:27.973159 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:27.973119 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-fe63ac/rate-test-model-fe63ac-kserve-54cb8c857c-lntw7"] Nov 03 00:15:27.976508 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:27.976469 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-fe63ac/rate-test-model-fe63ac-kserve-54cb8c857c-lntw7"] Nov 03 00:15:29.147878 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:29.147843 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af4882d8-765e-4167-9a4a-06a7d77f7437" path="/var/lib/kubelet/pods/af4882d8-765e-4167-9a4a-06a7d77f7437/volumes" Nov 03 00:15:29.617169 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:29.617133 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5"] Nov 03 00:15:29.617408 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:29.617383 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" podUID="51ba5367-449d-4292-9868-6ff46c0f94d3" containerName="istio-proxy" containerID="cri-o://e813b08762b3f2a4ca1fa2678ee9954751106248cc3d6b80dc12e38a4d26d578" gracePeriod=30 Nov 03 00:15:34.871209 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:34.871179 5076 generic.go:356] "Generic (PLEG): container finished" podID="51ba5367-449d-4292-9868-6ff46c0f94d3" containerID="e813b08762b3f2a4ca1fa2678ee9954751106248cc3d6b80dc12e38a4d26d578" exitCode=0 Nov 03 00:15:34.871616 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:34.871251 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" event={"ID":"51ba5367-449d-4292-9868-6ff46c0f94d3","Type":"ContainerDied","Data":"e813b08762b3f2a4ca1fa2678ee9954751106248cc3d6b80dc12e38a4d26d578"} Nov 03 00:15:34.871616 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:34.871308 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" event={"ID":"51ba5367-449d-4292-9868-6ff46c0f94d3","Type":"ContainerDied","Data":"b71d144c4443c05eed648d0b63dfc647a0ec440c0490314be19d387c4d586f7a"} Nov 03 00:15:34.871616 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:34.871331 5076 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b71d144c4443c05eed648d0b63dfc647a0ec440c0490314be19d387c4d586f7a" Nov 03 00:15:34.871616 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:34.871340 5076 scope.go:122] "RemoveContainer" containerID="7c99d708208871f9cca3b3e1f9608110d458c6fd09204a5433513332f9d7238f" Nov 03 00:15:34.884745 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:34.884720 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" Nov 03 00:15:35.057767 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:35.057727 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51ba5367-449d-4292-9868-6ff46c0f94d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51ba5367-449d-4292-9868-6ff46c0f94d3-workload-certs\") pod \"51ba5367-449d-4292-9868-6ff46c0f94d3\" (UID: \"51ba5367-449d-4292-9868-6ff46c0f94d3\") " Nov 03 00:15:35.057978 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:35.057803 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51ba5367-449d-4292-9868-6ff46c0f94d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ba5367-449d-4292-9868-6ff46c0f94d3-workload-socket\") pod \"51ba5367-449d-4292-9868-6ff46c0f94d3\" (UID: \"51ba5367-449d-4292-9868-6ff46c0f94d3\") " Nov 03 00:15:35.057978 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:35.057874 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51ba5367-449d-4292-9868-6ff46c0f94d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51ba5367-449d-4292-9868-6ff46c0f94d3-istio-podinfo\") pod \"51ba5367-449d-4292-9868-6ff46c0f94d3\" (UID: \"51ba5367-449d-4292-9868-6ff46c0f94d3\") " Nov 03 00:15:35.057978 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:35.057936 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51ba5367-449d-4292-9868-6ff46c0f94d3-kube-api-access-fd9ln\" (UniqueName: \"kubernetes.io/projected/51ba5367-449d-4292-9868-6ff46c0f94d3-kube-api-access-fd9ln\") pod \"51ba5367-449d-4292-9868-6ff46c0f94d3\" (UID: \"51ba5367-449d-4292-9868-6ff46c0f94d3\") " Nov 03 00:15:35.058207 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:35.057977 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51ba5367-449d-4292-9868-6ff46c0f94d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51ba5367-449d-4292-9868-6ff46c0f94d3-istiod-ca-cert\") pod \"51ba5367-449d-4292-9868-6ff46c0f94d3\" (UID: \"51ba5367-449d-4292-9868-6ff46c0f94d3\") " Nov 03 00:15:35.058207 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:35.058024 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51ba5367-449d-4292-9868-6ff46c0f94d3-workload-certs" pod "51ba5367-449d-4292-9868-6ff46c0f94d3" (UID: "51ba5367-449d-4292-9868-6ff46c0f94d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:15:35.058207 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:35.058066 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51ba5367-449d-4292-9868-6ff46c0f94d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51ba5367-449d-4292-9868-6ff46c0f94d3-istio-data\") pod \"51ba5367-449d-4292-9868-6ff46c0f94d3\" (UID: \"51ba5367-449d-4292-9868-6ff46c0f94d3\") " Nov 03 00:15:35.058207 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:35.058123 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51ba5367-449d-4292-9868-6ff46c0f94d3-istio-token\" (UniqueName: \"kubernetes.io/projected/51ba5367-449d-4292-9868-6ff46c0f94d3-istio-token\") pod \"51ba5367-449d-4292-9868-6ff46c0f94d3\" (UID: \"51ba5367-449d-4292-9868-6ff46c0f94d3\") " Nov 03 00:15:35.058207 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:35.058151 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51ba5367-449d-4292-9868-6ff46c0f94d3-workload-socket" pod "51ba5367-449d-4292-9868-6ff46c0f94d3" (UID: "51ba5367-449d-4292-9868-6ff46c0f94d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:15:35.058207 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:35.058174 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51ba5367-449d-4292-9868-6ff46c0f94d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ba5367-449d-4292-9868-6ff46c0f94d3-credential-socket\") pod \"51ba5367-449d-4292-9868-6ff46c0f94d3\" (UID: \"51ba5367-449d-4292-9868-6ff46c0f94d3\") " Nov 03 00:15:35.058543 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:35.058264 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51ba5367-449d-4292-9868-6ff46c0f94d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51ba5367-449d-4292-9868-6ff46c0f94d3-istio-envoy\") pod \"51ba5367-449d-4292-9868-6ff46c0f94d3\" (UID: \"51ba5367-449d-4292-9868-6ff46c0f94d3\") " Nov 03 00:15:35.058543 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:35.058347 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51ba5367-449d-4292-9868-6ff46c0f94d3-istio-data" pod "51ba5367-449d-4292-9868-6ff46c0f94d3" (UID: "51ba5367-449d-4292-9868-6ff46c0f94d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:15:35.058543 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:35.058484 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51ba5367-449d-4292-9868-6ff46c0f94d3-credential-socket" pod "51ba5367-449d-4292-9868-6ff46c0f94d3" (UID: "51ba5367-449d-4292-9868-6ff46c0f94d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:15:35.058543 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:35.058521 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51ba5367-449d-4292-9868-6ff46c0f94d3-istiod-ca-cert" pod "51ba5367-449d-4292-9868-6ff46c0f94d3" (UID: "51ba5367-449d-4292-9868-6ff46c0f94d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:35.058784 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:35.058756 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51ba5367-449d-4292-9868-6ff46c0f94d3-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:35.058848 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:35.058783 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ba5367-449d-4292-9868-6ff46c0f94d3-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:35.058848 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:35.058801 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51ba5367-449d-4292-9868-6ff46c0f94d3-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:35.058848 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:35.058815 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ba5367-449d-4292-9868-6ff46c0f94d3-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:35.058848 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:35.058829 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51ba5367-449d-4292-9868-6ff46c0f94d3-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:35.060586 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:35.060559 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51ba5367-449d-4292-9868-6ff46c0f94d3-istio-token" pod "51ba5367-449d-4292-9868-6ff46c0f94d3" (UID: "51ba5367-449d-4292-9868-6ff46c0f94d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:35.060863 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:35.060838 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51ba5367-449d-4292-9868-6ff46c0f94d3-istio-envoy" pod "51ba5367-449d-4292-9868-6ff46c0f94d3" (UID: "51ba5367-449d-4292-9868-6ff46c0f94d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:15:35.060863 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:35.060845 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51ba5367-449d-4292-9868-6ff46c0f94d3-kube-api-access-fd9ln" pod "51ba5367-449d-4292-9868-6ff46c0f94d3" (UID: "51ba5367-449d-4292-9868-6ff46c0f94d3"). InnerVolumeSpecName "kube-api-access-fd9ln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:35.061017 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:35.060976 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51ba5367-449d-4292-9868-6ff46c0f94d3-istio-podinfo" pod "51ba5367-449d-4292-9868-6ff46c0f94d3" (UID: "51ba5367-449d-4292-9868-6ff46c0f94d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:15:35.160096 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:35.160068 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51ba5367-449d-4292-9868-6ff46c0f94d3-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:35.160096 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:35.160096 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51ba5367-449d-4292-9868-6ff46c0f94d3-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:35.160264 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:35.160109 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd9ln\" (UniqueName: \"kubernetes.io/projected/51ba5367-449d-4292-9868-6ff46c0f94d3-kube-api-access-fd9ln\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:35.160264 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:35.160124 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51ba5367-449d-4292-9868-6ff46c0f94d3-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:35.878262 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:35.878233 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5" Nov 03 00:15:36.150069 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:36.149975 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5"] Nov 03 00:15:36.152253 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:36.152227 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c5148c-openshift-default-5c7989b6b7-pgrt5"] Nov 03 00:15:37.147694 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:15:37.147663 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51ba5367-449d-4292-9868-6ff46c0f94d3" path="/var/lib/kubelet/pods/51ba5367-449d-4292-9868-6ff46c0f94d3/volumes" Nov 03 00:15:39.884533 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:15:39.884494 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:16:09.883618 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:16:09.883518 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:16:23.040755 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:23.040717 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s"] Nov 03 00:16:23.041270 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:23.040946 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s" podUID="272994cd-ddb4-40ae-8cdd-5a91656a7603" containerName="maas-api" containerID="cri-o://408093d5816067a69aa16ffc85e6fa0d12e3764989350ea7bc7b24d67e54134d" gracePeriod=30 Nov 03 00:16:23.159293 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:23.159263 5076 generic.go:356] "Generic (PLEG): container finished" podID="272994cd-ddb4-40ae-8cdd-5a91656a7603" containerID="408093d5816067a69aa16ffc85e6fa0d12e3764989350ea7bc7b24d67e54134d" exitCode=0 Nov 03 00:16:23.159417 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:23.159338 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s" event={"ID":"272994cd-ddb4-40ae-8cdd-5a91656a7603","Type":"ContainerDied","Data":"408093d5816067a69aa16ffc85e6fa0d12e3764989350ea7bc7b24d67e54134d"} Nov 03 00:16:23.312828 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:23.312802 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s" Nov 03 00:16:23.443048 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:23.443016 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/272994cd-ddb4-40ae-8cdd-5a91656a7603-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/272994cd-ddb4-40ae-8cdd-5a91656a7603-maas-api-tls\") pod \"272994cd-ddb4-40ae-8cdd-5a91656a7603\" (UID: \"272994cd-ddb4-40ae-8cdd-5a91656a7603\") " Nov 03 00:16:23.443235 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:23.443091 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/272994cd-ddb4-40ae-8cdd-5a91656a7603-kube-api-access-cn4c9\" (UniqueName: \"kubernetes.io/projected/272994cd-ddb4-40ae-8cdd-5a91656a7603-kube-api-access-cn4c9\") pod \"272994cd-ddb4-40ae-8cdd-5a91656a7603\" (UID: \"272994cd-ddb4-40ae-8cdd-5a91656a7603\") " Nov 03 00:16:23.443235 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:23.443177 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/272994cd-ddb4-40ae-8cdd-5a91656a7603-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/272994cd-ddb4-40ae-8cdd-5a91656a7603-openshift-service-ca\") pod \"272994cd-ddb4-40ae-8cdd-5a91656a7603\" (UID: \"272994cd-ddb4-40ae-8cdd-5a91656a7603\") " Nov 03 00:16:23.443235 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:23.443208 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/272994cd-ddb4-40ae-8cdd-5a91656a7603-metrics-certs\" (UniqueName: \"kubernetes.io/secret/272994cd-ddb4-40ae-8cdd-5a91656a7603-metrics-certs\") pod \"272994cd-ddb4-40ae-8cdd-5a91656a7603\" (UID: \"272994cd-ddb4-40ae-8cdd-5a91656a7603\") " Nov 03 00:16:23.443918 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:23.443866 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/272994cd-ddb4-40ae-8cdd-5a91656a7603-openshift-service-ca" pod "272994cd-ddb4-40ae-8cdd-5a91656a7603" (UID: "272994cd-ddb4-40ae-8cdd-5a91656a7603"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:16:23.445393 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:23.445362 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/272994cd-ddb4-40ae-8cdd-5a91656a7603-kube-api-access-cn4c9" pod "272994cd-ddb4-40ae-8cdd-5a91656a7603" (UID: "272994cd-ddb4-40ae-8cdd-5a91656a7603"). InnerVolumeSpecName "kube-api-access-cn4c9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:16:23.445755 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:23.445733 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/272994cd-ddb4-40ae-8cdd-5a91656a7603-maas-api-tls" pod "272994cd-ddb4-40ae-8cdd-5a91656a7603" (UID: "272994cd-ddb4-40ae-8cdd-5a91656a7603"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:16:23.445843 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:23.445769 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/272994cd-ddb4-40ae-8cdd-5a91656a7603-metrics-certs" pod "272994cd-ddb4-40ae-8cdd-5a91656a7603" (UID: "272994cd-ddb4-40ae-8cdd-5a91656a7603"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:16:23.546931 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:23.546813 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/272994cd-ddb4-40ae-8cdd-5a91656a7603-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:23.546931 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:23.546862 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn4c9\" (UniqueName: \"kubernetes.io/projected/272994cd-ddb4-40ae-8cdd-5a91656a7603-kube-api-access-cn4c9\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:23.546931 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:23.546880 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/272994cd-ddb4-40ae-8cdd-5a91656a7603-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:23.546931 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:23.546910 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/272994cd-ddb4-40ae-8cdd-5a91656a7603-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:24.175621 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:24.175583 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s" event={"ID":"272994cd-ddb4-40ae-8cdd-5a91656a7603","Type":"ContainerDied","Data":"e5fe0a67474e9c4866e6940962d96ad18d53a9cae15bfaf77e506c3c6970d483"} Nov 03 00:16:24.176190 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:24.175659 5076 scope.go:122] "RemoveContainer" containerID="408093d5816067a69aa16ffc85e6fa0d12e3764989350ea7bc7b24d67e54134d" Nov 03 00:16:24.176190 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:24.175655 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s" Nov 03 00:16:24.347716 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:24.347681 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s"] Nov 03 00:16:24.351062 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:24.351023 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbec82-9b7d8bc58-m5j7s"] Nov 03 00:16:25.148174 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:25.148142 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="272994cd-ddb4-40ae-8cdd-5a91656a7603" path="/var/lib/kubelet/pods/272994cd-ddb4-40ae-8cdd-5a91656a7603/volumes" Nov 03 00:16:28.734185 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:28.734150 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr"] Nov 03 00:16:28.734609 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:28.734374 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" podUID="4deff109-12f0-42e8-8908-508e7f6eec68" containerName="istio-proxy" containerID="cri-o://da3b04c80a5932782a81cab47be5b6c880040878182c7a1ba0f9b538dc564ca3" gracePeriod=30 Nov 03 00:16:32.839140 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:32.839031 5076 scope.go:122] "RemoveContainer" containerID="e813b08762b3f2a4ca1fa2678ee9954751106248cc3d6b80dc12e38a4d26d578" Nov 03 00:16:34.014483 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.014411 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" Nov 03 00:16:34.060645 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.060614 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4deff109-12f0-42e8-8908-508e7f6eec68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4deff109-12f0-42e8-8908-508e7f6eec68-istio-podinfo\") pod \"4deff109-12f0-42e8-8908-508e7f6eec68\" (UID: \"4deff109-12f0-42e8-8908-508e7f6eec68\") " Nov 03 00:16:34.060816 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.060676 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4deff109-12f0-42e8-8908-508e7f6eec68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4deff109-12f0-42e8-8908-508e7f6eec68-workload-certs\") pod \"4deff109-12f0-42e8-8908-508e7f6eec68\" (UID: \"4deff109-12f0-42e8-8908-508e7f6eec68\") " Nov 03 00:16:34.060816 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.060702 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4deff109-12f0-42e8-8908-508e7f6eec68-kube-api-access-tgnpt\" (UniqueName: \"kubernetes.io/projected/4deff109-12f0-42e8-8908-508e7f6eec68-kube-api-access-tgnpt\") pod \"4deff109-12f0-42e8-8908-508e7f6eec68\" (UID: \"4deff109-12f0-42e8-8908-508e7f6eec68\") " Nov 03 00:16:34.060816 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.060772 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4deff109-12f0-42e8-8908-508e7f6eec68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4deff109-12f0-42e8-8908-508e7f6eec68-istio-envoy\") pod \"4deff109-12f0-42e8-8908-508e7f6eec68\" (UID: \"4deff109-12f0-42e8-8908-508e7f6eec68\") " Nov 03 00:16:34.060816 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.060792 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4deff109-12f0-42e8-8908-508e7f6eec68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4deff109-12f0-42e8-8908-508e7f6eec68-credential-socket\") pod \"4deff109-12f0-42e8-8908-508e7f6eec68\" (UID: \"4deff109-12f0-42e8-8908-508e7f6eec68\") " Nov 03 00:16:34.061044 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.060821 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4deff109-12f0-42e8-8908-508e7f6eec68-istio-token\" (UniqueName: \"kubernetes.io/projected/4deff109-12f0-42e8-8908-508e7f6eec68-istio-token\") pod \"4deff109-12f0-42e8-8908-508e7f6eec68\" (UID: \"4deff109-12f0-42e8-8908-508e7f6eec68\") " Nov 03 00:16:34.061044 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.060854 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4deff109-12f0-42e8-8908-508e7f6eec68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4deff109-12f0-42e8-8908-508e7f6eec68-istio-data\") pod \"4deff109-12f0-42e8-8908-508e7f6eec68\" (UID: \"4deff109-12f0-42e8-8908-508e7f6eec68\") " Nov 03 00:16:34.061044 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.060919 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4deff109-12f0-42e8-8908-508e7f6eec68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4deff109-12f0-42e8-8908-508e7f6eec68-istiod-ca-cert\") pod \"4deff109-12f0-42e8-8908-508e7f6eec68\" (UID: \"4deff109-12f0-42e8-8908-508e7f6eec68\") " Nov 03 00:16:34.061044 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.060947 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4deff109-12f0-42e8-8908-508e7f6eec68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4deff109-12f0-42e8-8908-508e7f6eec68-workload-socket\") pod \"4deff109-12f0-42e8-8908-508e7f6eec68\" (UID: \"4deff109-12f0-42e8-8908-508e7f6eec68\") " Nov 03 00:16:34.061277 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.061054 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4deff109-12f0-42e8-8908-508e7f6eec68-workload-certs" pod "4deff109-12f0-42e8-8908-508e7f6eec68" (UID: "4deff109-12f0-42e8-8908-508e7f6eec68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:34.061277 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.061151 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4deff109-12f0-42e8-8908-508e7f6eec68-credential-socket" pod "4deff109-12f0-42e8-8908-508e7f6eec68" (UID: "4deff109-12f0-42e8-8908-508e7f6eec68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:34.061392 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.061346 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4deff109-12f0-42e8-8908-508e7f6eec68-workload-socket" pod "4deff109-12f0-42e8-8908-508e7f6eec68" (UID: "4deff109-12f0-42e8-8908-508e7f6eec68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:34.061529 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.061411 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4deff109-12f0-42e8-8908-508e7f6eec68-istiod-ca-cert" pod "4deff109-12f0-42e8-8908-508e7f6eec68" (UID: "4deff109-12f0-42e8-8908-508e7f6eec68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:16:34.061529 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.061504 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4deff109-12f0-42e8-8908-508e7f6eec68-istio-data" pod "4deff109-12f0-42e8-8908-508e7f6eec68" (UID: "4deff109-12f0-42e8-8908-508e7f6eec68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:34.061529 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.061519 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4deff109-12f0-42e8-8908-508e7f6eec68-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:34.061665 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.061551 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4deff109-12f0-42e8-8908-508e7f6eec68-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:34.061665 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.061572 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4deff109-12f0-42e8-8908-508e7f6eec68-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:34.061665 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.061587 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4deff109-12f0-42e8-8908-508e7f6eec68-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:34.063062 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.063040 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4deff109-12f0-42e8-8908-508e7f6eec68-kube-api-access-tgnpt" pod "4deff109-12f0-42e8-8908-508e7f6eec68" (UID: "4deff109-12f0-42e8-8908-508e7f6eec68"). InnerVolumeSpecName "kube-api-access-tgnpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:16:34.063164 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.063062 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4deff109-12f0-42e8-8908-508e7f6eec68-istio-envoy" pod "4deff109-12f0-42e8-8908-508e7f6eec68" (UID: "4deff109-12f0-42e8-8908-508e7f6eec68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:34.063279 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.063257 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4deff109-12f0-42e8-8908-508e7f6eec68-istio-podinfo" pod "4deff109-12f0-42e8-8908-508e7f6eec68" (UID: "4deff109-12f0-42e8-8908-508e7f6eec68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:16:34.063444 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.063419 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4deff109-12f0-42e8-8908-508e7f6eec68-istio-token" pod "4deff109-12f0-42e8-8908-508e7f6eec68" (UID: "4deff109-12f0-42e8-8908-508e7f6eec68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:16:34.162196 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.162167 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4deff109-12f0-42e8-8908-508e7f6eec68-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:34.162196 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.162192 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgnpt\" (UniqueName: \"kubernetes.io/projected/4deff109-12f0-42e8-8908-508e7f6eec68-kube-api-access-tgnpt\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:34.162196 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.162203 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4deff109-12f0-42e8-8908-508e7f6eec68-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:34.162423 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.162212 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4deff109-12f0-42e8-8908-508e7f6eec68-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:34.162423 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.162222 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4deff109-12f0-42e8-8908-508e7f6eec68-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:34.236511 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.236473 5076 generic.go:356] "Generic (PLEG): container finished" podID="4deff109-12f0-42e8-8908-508e7f6eec68" containerID="da3b04c80a5932782a81cab47be5b6c880040878182c7a1ba0f9b538dc564ca3" exitCode=0 Nov 03 00:16:34.236671 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.236533 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" Nov 03 00:16:34.236671 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.236556 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" event={"ID":"4deff109-12f0-42e8-8908-508e7f6eec68","Type":"ContainerDied","Data":"da3b04c80a5932782a81cab47be5b6c880040878182c7a1ba0f9b538dc564ca3"} Nov 03 00:16:34.236671 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.236623 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr" event={"ID":"4deff109-12f0-42e8-8908-508e7f6eec68","Type":"ContainerDied","Data":"9aa2c3e29b72bf9f9b6c67ce3545f5f322540425745f742d0166f0ad53d20566"} Nov 03 00:16:34.236671 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.236650 5076 scope.go:122] "RemoveContainer" containerID="da3b04c80a5932782a81cab47be5b6c880040878182c7a1ba0f9b538dc564ca3" Nov 03 00:16:34.247365 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.247346 5076 scope.go:122] "RemoveContainer" containerID="e64290c914e0b51b805d0321fa2662f5f72e4e7b44f8e746614271b05e4e5b50" Nov 03 00:16:34.256363 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.256344 5076 scope.go:122] "RemoveContainer" containerID="da3b04c80a5932782a81cab47be5b6c880040878182c7a1ba0f9b538dc564ca3" Nov 03 00:16:34.256646 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:16:34.256627 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da3b04c80a5932782a81cab47be5b6c880040878182c7a1ba0f9b538dc564ca3\": container with ID starting with da3b04c80a5932782a81cab47be5b6c880040878182c7a1ba0f9b538dc564ca3 not found: ID does not exist" containerID="da3b04c80a5932782a81cab47be5b6c880040878182c7a1ba0f9b538dc564ca3" Nov 03 00:16:34.256687 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.256657 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da3b04c80a5932782a81cab47be5b6c880040878182c7a1ba0f9b538dc564ca3"} err="failed to get container status \"da3b04c80a5932782a81cab47be5b6c880040878182c7a1ba0f9b538dc564ca3\": rpc error: code = NotFound desc = could not find container \"da3b04c80a5932782a81cab47be5b6c880040878182c7a1ba0f9b538dc564ca3\": container with ID starting with da3b04c80a5932782a81cab47be5b6c880040878182c7a1ba0f9b538dc564ca3 not found: ID does not exist" Nov 03 00:16:34.256687 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.256679 5076 scope.go:122] "RemoveContainer" containerID="e64290c914e0b51b805d0321fa2662f5f72e4e7b44f8e746614271b05e4e5b50" Nov 03 00:16:34.256962 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:16:34.256943 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e64290c914e0b51b805d0321fa2662f5f72e4e7b44f8e746614271b05e4e5b50\": container with ID starting with e64290c914e0b51b805d0321fa2662f5f72e4e7b44f8e746614271b05e4e5b50 not found: ID does not exist" containerID="e64290c914e0b51b805d0321fa2662f5f72e4e7b44f8e746614271b05e4e5b50" Nov 03 00:16:34.257011 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.256971 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e64290c914e0b51b805d0321fa2662f5f72e4e7b44f8e746614271b05e4e5b50"} err="failed to get container status \"e64290c914e0b51b805d0321fa2662f5f72e4e7b44f8e746614271b05e4e5b50\": rpc error: code = NotFound desc = could not find container \"e64290c914e0b51b805d0321fa2662f5f72e4e7b44f8e746614271b05e4e5b50\": container with ID starting with e64290c914e0b51b805d0321fa2662f5f72e4e7b44f8e746614271b05e4e5b50 not found: ID does not exist" Nov 03 00:16:34.393541 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.393455 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr"] Nov 03 00:16:34.400847 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:34.400818 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fe63ac-openshift-default-7c9d6656f-ttbxr"] Nov 03 00:16:35.148740 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:35.148704 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4deff109-12f0-42e8-8908-508e7f6eec68" path="/var/lib/kubelet/pods/4deff109-12f0-42e8-8908-508e7f6eec68/volumes" Nov 03 00:16:35.684823 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:35.684790 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-28f82c/sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s"] Nov 03 00:16:39.882956 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:16:39.882916 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:16:50.705838 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:50.705790 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-28f82c/sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s" podUID="d3c9fa49-400c-4d00-bc52-cfc65b24d3d3" containerName="main" containerID="cri-o://76b7fcb50a6d4959efcc22792a02621501ef2b3abf5f8ca0275851e920f04af7" gracePeriod=45 Nov 03 00:16:50.972859 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:50.972831 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-28f82c/sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s" Nov 03 00:16:51.050662 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:51.050630 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-tmp-dir\") pod \"d3c9fa49-400c-4d00-bc52-cfc65b24d3d3\" (UID: \"d3c9fa49-400c-4d00-bc52-cfc65b24d3d3\") " Nov 03 00:16:51.050877 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:51.050701 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-kube-api-access-htlt6\" (UniqueName: \"kubernetes.io/projected/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-kube-api-access-htlt6\") pod \"d3c9fa49-400c-4d00-bc52-cfc65b24d3d3\" (UID: \"d3c9fa49-400c-4d00-bc52-cfc65b24d3d3\") " Nov 03 00:16:51.050877 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:51.050742 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-home\" (UniqueName: \"kubernetes.io/empty-dir/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-home\") pod \"d3c9fa49-400c-4d00-bc52-cfc65b24d3d3\" (UID: \"d3c9fa49-400c-4d00-bc52-cfc65b24d3d3\") " Nov 03 00:16:51.050877 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:51.050809 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-model-cache\") pod \"d3c9fa49-400c-4d00-bc52-cfc65b24d3d3\" (UID: \"d3c9fa49-400c-4d00-bc52-cfc65b24d3d3\") " Nov 03 00:16:51.050877 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:51.050873 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-tls-certs\") pod \"d3c9fa49-400c-4d00-bc52-cfc65b24d3d3\" (UID: \"d3c9fa49-400c-4d00-bc52-cfc65b24d3d3\") " Nov 03 00:16:51.051131 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:51.050967 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-tmp-dir" pod "d3c9fa49-400c-4d00-bc52-cfc65b24d3d3" (UID: "d3c9fa49-400c-4d00-bc52-cfc65b24d3d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:51.051131 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:51.051023 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-dshm\") pod \"d3c9fa49-400c-4d00-bc52-cfc65b24d3d3\" (UID: \"d3c9fa49-400c-4d00-bc52-cfc65b24d3d3\") " Nov 03 00:16:51.051131 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:51.051038 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-home" pod "d3c9fa49-400c-4d00-bc52-cfc65b24d3d3" (UID: "d3c9fa49-400c-4d00-bc52-cfc65b24d3d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:51.051131 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:51.051106 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-model-cache" pod "d3c9fa49-400c-4d00-bc52-cfc65b24d3d3" (UID: "d3c9fa49-400c-4d00-bc52-cfc65b24d3d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:51.051444 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:51.051425 5076 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-tmp-dir\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:51.051514 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:51.051450 5076 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-home\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:51.051514 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:51.051464 5076 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-model-cache\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:51.052997 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:51.052966 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-dshm" pod "d3c9fa49-400c-4d00-bc52-cfc65b24d3d3" (UID: "d3c9fa49-400c-4d00-bc52-cfc65b24d3d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:51.053103 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:51.053004 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-kube-api-access-htlt6" pod "d3c9fa49-400c-4d00-bc52-cfc65b24d3d3" (UID: "d3c9fa49-400c-4d00-bc52-cfc65b24d3d3"). InnerVolumeSpecName "kube-api-access-htlt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:16:51.053103 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:51.053036 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-tls-certs" pod "d3c9fa49-400c-4d00-bc52-cfc65b24d3d3" (UID: "d3c9fa49-400c-4d00-bc52-cfc65b24d3d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:16:51.152557 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:51.152523 5076 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-tls-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:51.152557 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:51.152553 5076 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-dshm\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:51.152557 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:51.152564 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htlt6\" (UniqueName: \"kubernetes.io/projected/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3-kube-api-access-htlt6\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:51.336120 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:51.336039 5076 generic.go:356] "Generic (PLEG): container finished" podID="d3c9fa49-400c-4d00-bc52-cfc65b24d3d3" containerID="76b7fcb50a6d4959efcc22792a02621501ef2b3abf5f8ca0275851e920f04af7" exitCode=0 Nov 03 00:16:51.336120 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:51.336089 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-28f82c/sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s" Nov 03 00:16:51.336305 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:51.336122 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-28f82c/sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s" event={"ID":"d3c9fa49-400c-4d00-bc52-cfc65b24d3d3","Type":"ContainerDied","Data":"76b7fcb50a6d4959efcc22792a02621501ef2b3abf5f8ca0275851e920f04af7"} Nov 03 00:16:51.336305 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:51.336166 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-28f82c/sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s" event={"ID":"d3c9fa49-400c-4d00-bc52-cfc65b24d3d3","Type":"ContainerDied","Data":"391b265896af73ed81195b31f3b719a71172094744861b79a28bd9bd68c347f5"} Nov 03 00:16:51.336305 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:51.336182 5076 scope.go:122] "RemoveContainer" containerID="76b7fcb50a6d4959efcc22792a02621501ef2b3abf5f8ca0275851e920f04af7" Nov 03 00:16:51.346266 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:51.346238 5076 scope.go:122] "RemoveContainer" containerID="76b7fcb50a6d4959efcc22792a02621501ef2b3abf5f8ca0275851e920f04af7" Nov 03 00:16:51.346612 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:16:51.346589 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76b7fcb50a6d4959efcc22792a02621501ef2b3abf5f8ca0275851e920f04af7\": container with ID starting with 76b7fcb50a6d4959efcc22792a02621501ef2b3abf5f8ca0275851e920f04af7 not found: ID does not exist" containerID="76b7fcb50a6d4959efcc22792a02621501ef2b3abf5f8ca0275851e920f04af7" Nov 03 00:16:51.346705 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:51.346623 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76b7fcb50a6d4959efcc22792a02621501ef2b3abf5f8ca0275851e920f04af7"} err="failed to get container status \"76b7fcb50a6d4959efcc22792a02621501ef2b3abf5f8ca0275851e920f04af7\": rpc error: code = NotFound desc = could not find container \"76b7fcb50a6d4959efcc22792a02621501ef2b3abf5f8ca0275851e920f04af7\": container with ID starting with 76b7fcb50a6d4959efcc22792a02621501ef2b3abf5f8ca0275851e920f04af7 not found: ID does not exist" Nov 03 00:16:51.492666 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:51.492636 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-28f82c/sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s"] Nov 03 00:16:51.539831 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:51.539798 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-28f82c/sub-test-model-28f82c-kserve-8f96d6d5f-mtf7s"] Nov 03 00:16:53.148265 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:16:53.148228 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3c9fa49-400c-4d00-bc52-cfc65b24d3d3" path="/var/lib/kubelet/pods/d3c9fa49-400c-4d00-bc52-cfc65b24d3d3/volumes" Nov 03 00:17:09.884397 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:17:09.884357 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:17:14.805239 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:14.805146 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq"] Nov 03 00:17:14.805731 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:14.805345 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq" podUID="62d85249-11d4-47a6-955d-978815d12fd9" containerName="maas-api" containerID="cri-o://6de065b514eeccb4211af4e0fb0421669f61a0f8c28d1c076aa990ecd668029c" gracePeriod=30 Nov 03 00:17:15.085798 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:15.085767 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq" Nov 03 00:17:15.149173 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:15.149135 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62d85249-11d4-47a6-955d-978815d12fd9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62d85249-11d4-47a6-955d-978815d12fd9-maas-api-tls\") pod \"62d85249-11d4-47a6-955d-978815d12fd9\" (UID: \"62d85249-11d4-47a6-955d-978815d12fd9\") " Nov 03 00:17:15.149422 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:15.149199 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62d85249-11d4-47a6-955d-978815d12fd9-kube-api-access-rbvb9\" (UniqueName: \"kubernetes.io/projected/62d85249-11d4-47a6-955d-978815d12fd9-kube-api-access-rbvb9\") pod \"62d85249-11d4-47a6-955d-978815d12fd9\" (UID: \"62d85249-11d4-47a6-955d-978815d12fd9\") " Nov 03 00:17:15.149422 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:15.149273 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62d85249-11d4-47a6-955d-978815d12fd9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62d85249-11d4-47a6-955d-978815d12fd9-openshift-service-ca\") pod \"62d85249-11d4-47a6-955d-978815d12fd9\" (UID: \"62d85249-11d4-47a6-955d-978815d12fd9\") " Nov 03 00:17:15.149422 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:15.149309 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62d85249-11d4-47a6-955d-978815d12fd9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/62d85249-11d4-47a6-955d-978815d12fd9-metrics-certs\") pod \"62d85249-11d4-47a6-955d-978815d12fd9\" (UID: \"62d85249-11d4-47a6-955d-978815d12fd9\") " Nov 03 00:17:15.152617 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:15.152587 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62d85249-11d4-47a6-955d-978815d12fd9-openshift-service-ca" pod "62d85249-11d4-47a6-955d-978815d12fd9" (UID: "62d85249-11d4-47a6-955d-978815d12fd9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:17:15.152826 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:15.152794 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62d85249-11d4-47a6-955d-978815d12fd9-metrics-certs" pod "62d85249-11d4-47a6-955d-978815d12fd9" (UID: "62d85249-11d4-47a6-955d-978815d12fd9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:17:15.154677 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:15.154652 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62d85249-11d4-47a6-955d-978815d12fd9-maas-api-tls" pod "62d85249-11d4-47a6-955d-978815d12fd9" (UID: "62d85249-11d4-47a6-955d-978815d12fd9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:17:15.155436 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:15.155412 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62d85249-11d4-47a6-955d-978815d12fd9-kube-api-access-rbvb9" pod "62d85249-11d4-47a6-955d-978815d12fd9" (UID: "62d85249-11d4-47a6-955d-978815d12fd9"). InnerVolumeSpecName "kube-api-access-rbvb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:17:15.251384 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:15.251350 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62d85249-11d4-47a6-955d-978815d12fd9-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:15.251384 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:15.251379 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbvb9\" (UniqueName: \"kubernetes.io/projected/62d85249-11d4-47a6-955d-978815d12fd9-kube-api-access-rbvb9\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:15.251384 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:15.251391 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62d85249-11d4-47a6-955d-978815d12fd9-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:15.251629 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:15.251400 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62d85249-11d4-47a6-955d-978815d12fd9-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:15.501100 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:15.501055 5076 generic.go:356] "Generic (PLEG): container finished" podID="62d85249-11d4-47a6-955d-978815d12fd9" containerID="6de065b514eeccb4211af4e0fb0421669f61a0f8c28d1c076aa990ecd668029c" exitCode=0 Nov 03 00:17:15.501279 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:15.501096 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq" event={"ID":"62d85249-11d4-47a6-955d-978815d12fd9","Type":"ContainerDied","Data":"6de065b514eeccb4211af4e0fb0421669f61a0f8c28d1c076aa990ecd668029c"} Nov 03 00:17:15.501279 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:15.501120 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq" Nov 03 00:17:15.501279 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:15.501155 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq" event={"ID":"62d85249-11d4-47a6-955d-978815d12fd9","Type":"ContainerDied","Data":"3902aab50cf16528aa9182160675cb538d03bdbd23d328f47a416a2a7144fc6a"} Nov 03 00:17:15.501279 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:15.501176 5076 scope.go:122] "RemoveContainer" containerID="6de065b514eeccb4211af4e0fb0421669f61a0f8c28d1c076aa990ecd668029c" Nov 03 00:17:15.512536 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:15.512512 5076 scope.go:122] "RemoveContainer" containerID="6de065b514eeccb4211af4e0fb0421669f61a0f8c28d1c076aa990ecd668029c" Nov 03 00:17:15.512842 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:17:15.512820 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6de065b514eeccb4211af4e0fb0421669f61a0f8c28d1c076aa990ecd668029c\": container with ID starting with 6de065b514eeccb4211af4e0fb0421669f61a0f8c28d1c076aa990ecd668029c not found: ID does not exist" containerID="6de065b514eeccb4211af4e0fb0421669f61a0f8c28d1c076aa990ecd668029c" Nov 03 00:17:15.512916 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:15.512852 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6de065b514eeccb4211af4e0fb0421669f61a0f8c28d1c076aa990ecd668029c"} err="failed to get container status \"6de065b514eeccb4211af4e0fb0421669f61a0f8c28d1c076aa990ecd668029c\": rpc error: code = NotFound desc = could not find container \"6de065b514eeccb4211af4e0fb0421669f61a0f8c28d1c076aa990ecd668029c\": container with ID starting with 6de065b514eeccb4211af4e0fb0421669f61a0f8c28d1c076aa990ecd668029c not found: ID does not exist" Nov 03 00:17:15.660037 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:15.659997 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq"] Nov 03 00:17:15.669748 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:15.669716 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-28f82c-6b89f58b7f-gngsq"] Nov 03 00:17:17.147994 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:17.147963 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62d85249-11d4-47a6-955d-978815d12fd9" path="/var/lib/kubelet/pods/62d85249-11d4-47a6-955d-978815d12fd9/volumes" Nov 03 00:17:23.553256 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.553222 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4"] Nov 03 00:17:23.553768 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.553684 5076 cpu_manager.go:414] "containerMap: removing container" podUID="51ba5367-449d-4292-9868-6ff46c0f94d3" containerName="istio-proxy" Nov 03 00:17:23.553768 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.553695 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51ba5367-449d-4292-9868-6ff46c0f94d3" containerName="istio-proxy" Nov 03 00:17:23.553768 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.553707 5076 cpu_manager.go:414] "containerMap: removing container" podUID="af4882d8-765e-4167-9a4a-06a7d77f7437" containerName="main" Nov 03 00:17:23.553768 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.553714 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af4882d8-765e-4167-9a4a-06a7d77f7437" containerName="main" Nov 03 00:17:23.553768 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.553728 5076 cpu_manager.go:414] "containerMap: removing container" podUID="62d85249-11d4-47a6-955d-978815d12fd9" containerName="maas-api" Nov 03 00:17:23.553768 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.553732 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62d85249-11d4-47a6-955d-978815d12fd9" containerName="maas-api" Nov 03 00:17:23.553768 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.553747 5076 cpu_manager.go:414] "containerMap: removing container" podUID="d3c9fa49-400c-4d00-bc52-cfc65b24d3d3" containerName="main" Nov 03 00:17:23.553768 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.553752 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3c9fa49-400c-4d00-bc52-cfc65b24d3d3" containerName="main" Nov 03 00:17:23.553768 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.553761 5076 cpu_manager.go:414] "containerMap: removing container" podUID="9e58f4ff-b1f9-45d4-95ad-e628431da7b7" containerName="cleanup" Nov 03 00:17:23.553768 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.553766 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e58f4ff-b1f9-45d4-95ad-e628431da7b7" containerName="cleanup" Nov 03 00:17:23.553768 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.553774 5076 cpu_manager.go:414] "containerMap: removing container" podUID="4d4868a3-1492-4d61-81ae-080156be2b73" containerName="cleanup" Nov 03 00:17:23.554209 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.553778 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d4868a3-1492-4d61-81ae-080156be2b73" containerName="cleanup" Nov 03 00:17:23.554209 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.553787 5076 cpu_manager.go:414] "containerMap: removing container" podUID="4deff109-12f0-42e8-8908-508e7f6eec68" containerName="istio-proxy" Nov 03 00:17:23.554209 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.553792 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4deff109-12f0-42e8-8908-508e7f6eec68" containerName="istio-proxy" Nov 03 00:17:23.554209 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.553799 5076 cpu_manager.go:414] "containerMap: removing container" podUID="9e58f4ff-b1f9-45d4-95ad-e628431da7b7" containerName="cleanup" Nov 03 00:17:23.554209 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.553803 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e58f4ff-b1f9-45d4-95ad-e628431da7b7" containerName="cleanup" Nov 03 00:17:23.554209 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.553811 5076 cpu_manager.go:414] "containerMap: removing container" podUID="4d4868a3-1492-4d61-81ae-080156be2b73" containerName="cleanup" Nov 03 00:17:23.554209 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.553815 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d4868a3-1492-4d61-81ae-080156be2b73" containerName="cleanup" Nov 03 00:17:23.554209 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.553827 5076 cpu_manager.go:414] "containerMap: removing container" podUID="4deff109-12f0-42e8-8908-508e7f6eec68" containerName="istio-proxy" Nov 03 00:17:23.554209 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.553831 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4deff109-12f0-42e8-8908-508e7f6eec68" containerName="istio-proxy" Nov 03 00:17:23.554209 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.553852 5076 cpu_manager.go:414] "containerMap: removing container" podUID="4d4868a3-1492-4d61-81ae-080156be2b73" containerName="cleanup" Nov 03 00:17:23.554209 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.553857 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d4868a3-1492-4d61-81ae-080156be2b73" containerName="cleanup" Nov 03 00:17:23.554209 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.553865 5076 cpu_manager.go:414] "containerMap: removing container" podUID="272994cd-ddb4-40ae-8cdd-5a91656a7603" containerName="maas-api" Nov 03 00:17:23.554209 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.553870 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="272994cd-ddb4-40ae-8cdd-5a91656a7603" containerName="maas-api" Nov 03 00:17:23.554209 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.553971 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3c9fa49-400c-4d00-bc52-cfc65b24d3d3" containerName="main" Nov 03 00:17:23.554209 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.553982 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e58f4ff-b1f9-45d4-95ad-e628431da7b7" containerName="cleanup" Nov 03 00:17:23.554209 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.553988 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d4868a3-1492-4d61-81ae-080156be2b73" containerName="cleanup" Nov 03 00:17:23.554209 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.553994 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="4deff109-12f0-42e8-8908-508e7f6eec68" containerName="istio-proxy" Nov 03 00:17:23.554209 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.553999 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e58f4ff-b1f9-45d4-95ad-e628431da7b7" containerName="cleanup" Nov 03 00:17:23.554209 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.554005 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d4868a3-1492-4d61-81ae-080156be2b73" containerName="cleanup" Nov 03 00:17:23.554209 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.554013 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="51ba5367-449d-4292-9868-6ff46c0f94d3" containerName="istio-proxy" Nov 03 00:17:23.554209 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.554018 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="272994cd-ddb4-40ae-8cdd-5a91656a7603" containerName="maas-api" Nov 03 00:17:23.554209 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.554023 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="51ba5367-449d-4292-9868-6ff46c0f94d3" containerName="istio-proxy" Nov 03 00:17:23.554209 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.554027 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="af4882d8-765e-4167-9a4a-06a7d77f7437" containerName="main" Nov 03 00:17:23.554209 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.554033 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="62d85249-11d4-47a6-955d-978815d12fd9" containerName="maas-api" Nov 03 00:17:23.558805 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.558786 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" Nov 03 00:17:23.609014 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.608981 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-ba1918-openshift-default-dockercfg-k6bc8\"" Nov 03 00:17:23.650859 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.650821 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6827543-3552-4c6c-bfed-203b1081cdc2-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4\" (UID: \"c6827543-3552-4c6c-bfed-203b1081cdc2\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" Nov 03 00:17:23.651056 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.650868 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6827543-3552-4c6c-bfed-203b1081cdc2-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4\" (UID: \"c6827543-3552-4c6c-bfed-203b1081cdc2\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" Nov 03 00:17:23.651056 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.651012 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6827543-3552-4c6c-bfed-203b1081cdc2-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4\" (UID: \"c6827543-3552-4c6c-bfed-203b1081cdc2\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" Nov 03 00:17:23.651056 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.651050 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6827543-3552-4c6c-bfed-203b1081cdc2-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4\" (UID: \"c6827543-3552-4c6c-bfed-203b1081cdc2\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" Nov 03 00:17:23.651270 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.651231 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6827543-3552-4c6c-bfed-203b1081cdc2-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4\" (UID: \"c6827543-3552-4c6c-bfed-203b1081cdc2\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" Nov 03 00:17:23.651329 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.651276 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6827543-3552-4c6c-bfed-203b1081cdc2-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4\" (UID: \"c6827543-3552-4c6c-bfed-203b1081cdc2\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" Nov 03 00:17:23.651431 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.651401 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6827543-3552-4c6c-bfed-203b1081cdc2-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4\" (UID: \"c6827543-3552-4c6c-bfed-203b1081cdc2\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" Nov 03 00:17:23.651501 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.651443 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6827543-3552-4c6c-bfed-203b1081cdc2-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4\" (UID: \"c6827543-3552-4c6c-bfed-203b1081cdc2\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" Nov 03 00:17:23.651501 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.651462 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb2vz\" (UniqueName: \"kubernetes.io/projected/c6827543-3552-4c6c-bfed-203b1081cdc2-kube-api-access-sb2vz\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4\" (UID: \"c6827543-3552-4c6c-bfed-203b1081cdc2\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" Nov 03 00:17:23.659303 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.659276 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4"] Nov 03 00:17:23.752594 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.752559 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6827543-3552-4c6c-bfed-203b1081cdc2-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4\" (UID: \"c6827543-3552-4c6c-bfed-203b1081cdc2\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" Nov 03 00:17:23.752594 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.752593 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6827543-3552-4c6c-bfed-203b1081cdc2-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4\" (UID: \"c6827543-3552-4c6c-bfed-203b1081cdc2\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" Nov 03 00:17:23.752851 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.752649 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6827543-3552-4c6c-bfed-203b1081cdc2-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4\" (UID: \"c6827543-3552-4c6c-bfed-203b1081cdc2\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" Nov 03 00:17:23.752851 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.752672 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6827543-3552-4c6c-bfed-203b1081cdc2-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4\" (UID: \"c6827543-3552-4c6c-bfed-203b1081cdc2\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" Nov 03 00:17:23.752851 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.752728 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6827543-3552-4c6c-bfed-203b1081cdc2-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4\" (UID: \"c6827543-3552-4c6c-bfed-203b1081cdc2\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" Nov 03 00:17:23.752851 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.752753 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6827543-3552-4c6c-bfed-203b1081cdc2-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4\" (UID: \"c6827543-3552-4c6c-bfed-203b1081cdc2\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" Nov 03 00:17:23.752851 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.752779 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb2vz\" (UniqueName: \"kubernetes.io/projected/c6827543-3552-4c6c-bfed-203b1081cdc2-kube-api-access-sb2vz\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4\" (UID: \"c6827543-3552-4c6c-bfed-203b1081cdc2\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" Nov 03 00:17:23.752851 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.752826 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6827543-3552-4c6c-bfed-203b1081cdc2-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4\" (UID: \"c6827543-3552-4c6c-bfed-203b1081cdc2\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" Nov 03 00:17:23.753216 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.753114 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6827543-3552-4c6c-bfed-203b1081cdc2-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4\" (UID: \"c6827543-3552-4c6c-bfed-203b1081cdc2\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" Nov 03 00:17:23.753216 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.753170 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6827543-3552-4c6c-bfed-203b1081cdc2-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4\" (UID: \"c6827543-3552-4c6c-bfed-203b1081cdc2\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" Nov 03 00:17:23.753216 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.753170 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6827543-3552-4c6c-bfed-203b1081cdc2-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4\" (UID: \"c6827543-3552-4c6c-bfed-203b1081cdc2\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" Nov 03 00:17:23.753400 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.753241 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6827543-3552-4c6c-bfed-203b1081cdc2-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4\" (UID: \"c6827543-3552-4c6c-bfed-203b1081cdc2\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" Nov 03 00:17:23.753400 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.753344 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6827543-3552-4c6c-bfed-203b1081cdc2-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4\" (UID: \"c6827543-3552-4c6c-bfed-203b1081cdc2\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" Nov 03 00:17:23.753802 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.753779 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6827543-3552-4c6c-bfed-203b1081cdc2-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4\" (UID: \"c6827543-3552-4c6c-bfed-203b1081cdc2\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" Nov 03 00:17:23.755125 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.755105 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6827543-3552-4c6c-bfed-203b1081cdc2-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4\" (UID: \"c6827543-3552-4c6c-bfed-203b1081cdc2\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" Nov 03 00:17:23.755376 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.755355 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6827543-3552-4c6c-bfed-203b1081cdc2-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4\" (UID: \"c6827543-3552-4c6c-bfed-203b1081cdc2\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" Nov 03 00:17:23.807034 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.806950 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6827543-3552-4c6c-bfed-203b1081cdc2-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4\" (UID: \"c6827543-3552-4c6c-bfed-203b1081cdc2\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" Nov 03 00:17:23.807185 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.807035 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb2vz\" (UniqueName: \"kubernetes.io/projected/c6827543-3552-4c6c-bfed-203b1081cdc2-kube-api-access-sb2vz\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4\" (UID: \"c6827543-3552-4c6c-bfed-203b1081cdc2\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" Nov 03 00:17:23.871614 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:23.871580 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" Nov 03 00:17:24.289246 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:24.289222 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4"] Nov 03 00:17:24.290707 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:17:24.290681 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6827543_3552_4c6c_bfed_203b1081cdc2.slice/crio-e46958704ce350372af5c360f0db3a5ef131560ea623d82415175f7f84d29974 WatchSource:0}: Error finding container e46958704ce350372af5c360f0db3a5ef131560ea623d82415175f7f84d29974: Status 404 returned error can't find the container with id e46958704ce350372af5c360f0db3a5ef131560ea623d82415175f7f84d29974 Nov 03 00:17:24.293035 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:24.293004 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:17:24.293129 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:24.293084 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:17:24.293129 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:24.293115 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:17:24.555560 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:24.555469 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" event={"ID":"c6827543-3552-4c6c-bfed-203b1081cdc2","Type":"ContainerStarted","Data":"30b6ca2cfbc73992668ca258d859d464214df7aad294058eeafeb63c24278cb7"} Nov 03 00:17:24.555560 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:24.555533 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" event={"ID":"c6827543-3552-4c6c-bfed-203b1081cdc2","Type":"ContainerStarted","Data":"e46958704ce350372af5c360f0db3a5ef131560ea623d82415175f7f84d29974"} Nov 03 00:17:24.669858 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:24.669801 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" podStartSLOduration=1.6697871480000002 podStartE2EDuration="1.669787148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:17:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:17:24.668942527 +0000 UTC m=+5093.637474311" watchObservedRunningTime="2026-11-03 00:17:24.669787148 +0000 UTC m=+5093.638318931" Nov 03 00:17:24.872376 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:24.872289 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" Nov 03 00:17:25.874181 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:25.874147 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Nov 03 00:17:25.874600 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:25.874211 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" podUID="c6827543-3552-4c6c-bfed-203b1081cdc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Nov 03 00:17:26.872604 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:26.872568 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Nov 03 00:17:26.872763 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:26.872620 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" podUID="c6827543-3552-4c6c-bfed-203b1081cdc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Nov 03 00:17:27.885312 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:27.881660 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Nov 03 00:17:27.885312 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:27.881719 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" podUID="c6827543-3552-4c6c-bfed-203b1081cdc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Nov 03 00:17:28.872585 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:28.872547 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Nov 03 00:17:28.876933 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:28.873181 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" podUID="c6827543-3552-4c6c-bfed-203b1081cdc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Nov 03 00:17:29.872710 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:29.872668 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Nov 03 00:17:29.873165 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:29.872722 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" podUID="c6827543-3552-4c6c-bfed-203b1081cdc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Nov 03 00:17:30.872395 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:30.872352 5076 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Nov 03 00:17:30.872578 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:30.872424 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" podUID="c6827543-3552-4c6c-bfed-203b1081cdc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Nov 03 00:17:31.803981 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:31.803942 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw"] Nov 03 00:17:31.804417 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:31.804134 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw" podUID="365ba2a8-0d25-4d3e-b6b0-eb577be48fcd" containerName="maas-api" containerID="cri-o://d15ef001caec8a7c00633e7f071acb6486374af60e922b63a650b3791a01421c" gracePeriod=30 Nov 03 00:17:32.088525 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:32.088494 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw" Nov 03 00:17:32.176664 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:32.176628 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" Nov 03 00:17:32.177123 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:32.177094 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" Nov 03 00:17:32.179467 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:32.179445 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" Nov 03 00:17:32.183562 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:32.183544 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 03 00:17:32.190089 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:32.190065 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/365ba2a8-0d25-4d3e-b6b0-eb577be48fcd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/365ba2a8-0d25-4d3e-b6b0-eb577be48fcd-metrics-certs\") pod \"365ba2a8-0d25-4d3e-b6b0-eb577be48fcd\" (UID: \"365ba2a8-0d25-4d3e-b6b0-eb577be48fcd\") " Nov 03 00:17:32.190266 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:32.190114 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/365ba2a8-0d25-4d3e-b6b0-eb577be48fcd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/365ba2a8-0d25-4d3e-b6b0-eb577be48fcd-maas-api-tls\") pod \"365ba2a8-0d25-4d3e-b6b0-eb577be48fcd\" (UID: \"365ba2a8-0d25-4d3e-b6b0-eb577be48fcd\") " Nov 03 00:17:32.190266 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:32.190171 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/365ba2a8-0d25-4d3e-b6b0-eb577be48fcd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/365ba2a8-0d25-4d3e-b6b0-eb577be48fcd-openshift-service-ca\") pod \"365ba2a8-0d25-4d3e-b6b0-eb577be48fcd\" (UID: \"365ba2a8-0d25-4d3e-b6b0-eb577be48fcd\") " Nov 03 00:17:32.190433 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:32.190262 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/365ba2a8-0d25-4d3e-b6b0-eb577be48fcd-kube-api-access-jf6ms\" (UniqueName: \"kubernetes.io/projected/365ba2a8-0d25-4d3e-b6b0-eb577be48fcd-kube-api-access-jf6ms\") pod \"365ba2a8-0d25-4d3e-b6b0-eb577be48fcd\" (UID: \"365ba2a8-0d25-4d3e-b6b0-eb577be48fcd\") " Nov 03 00:17:32.191433 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:32.191409 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/365ba2a8-0d25-4d3e-b6b0-eb577be48fcd-openshift-service-ca" pod "365ba2a8-0d25-4d3e-b6b0-eb577be48fcd" (UID: "365ba2a8-0d25-4d3e-b6b0-eb577be48fcd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:17:32.197038 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:32.197012 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/365ba2a8-0d25-4d3e-b6b0-eb577be48fcd-maas-api-tls" pod "365ba2a8-0d25-4d3e-b6b0-eb577be48fcd" (UID: "365ba2a8-0d25-4d3e-b6b0-eb577be48fcd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:17:32.197141 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:32.197032 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/365ba2a8-0d25-4d3e-b6b0-eb577be48fcd-metrics-certs" pod "365ba2a8-0d25-4d3e-b6b0-eb577be48fcd" (UID: "365ba2a8-0d25-4d3e-b6b0-eb577be48fcd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:17:32.197141 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:32.197051 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/365ba2a8-0d25-4d3e-b6b0-eb577be48fcd-kube-api-access-jf6ms" pod "365ba2a8-0d25-4d3e-b6b0-eb577be48fcd" (UID: "365ba2a8-0d25-4d3e-b6b0-eb577be48fcd"). InnerVolumeSpecName "kube-api-access-jf6ms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:17:32.283525 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:32.283489 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 03 00:17:32.291351 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:32.291311 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf6ms\" (UniqueName: \"kubernetes.io/projected/365ba2a8-0d25-4d3e-b6b0-eb577be48fcd-kube-api-access-jf6ms\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:32.291351 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:32.291346 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/365ba2a8-0d25-4d3e-b6b0-eb577be48fcd-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:32.291599 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:32.291361 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/365ba2a8-0d25-4d3e-b6b0-eb577be48fcd-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:32.291599 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:32.291375 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/365ba2a8-0d25-4d3e-b6b0-eb577be48fcd-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:32.620648 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:32.620608 5076 generic.go:356] "Generic (PLEG): container finished" podID="365ba2a8-0d25-4d3e-b6b0-eb577be48fcd" containerID="d15ef001caec8a7c00633e7f071acb6486374af60e922b63a650b3791a01421c" exitCode=0 Nov 03 00:17:32.620838 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:32.620676 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw" Nov 03 00:17:32.620838 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:32.620686 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw" event={"ID":"365ba2a8-0d25-4d3e-b6b0-eb577be48fcd","Type":"ContainerDied","Data":"d15ef001caec8a7c00633e7f071acb6486374af60e922b63a650b3791a01421c"} Nov 03 00:17:32.620838 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:32.620762 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw" event={"ID":"365ba2a8-0d25-4d3e-b6b0-eb577be48fcd","Type":"ContainerDied","Data":"412b10dd52afc6346502102ef3125058c21c59d0ddbca28a037970711c97057c"} Nov 03 00:17:32.620838 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:32.620786 5076 scope.go:122] "RemoveContainer" containerID="d15ef001caec8a7c00633e7f071acb6486374af60e922b63a650b3791a01421c" Nov 03 00:17:32.631456 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:32.631436 5076 scope.go:122] "RemoveContainer" containerID="d15ef001caec8a7c00633e7f071acb6486374af60e922b63a650b3791a01421c" Nov 03 00:17:32.631848 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:17:32.631821 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d15ef001caec8a7c00633e7f071acb6486374af60e922b63a650b3791a01421c\": container with ID starting with d15ef001caec8a7c00633e7f071acb6486374af60e922b63a650b3791a01421c not found: ID does not exist" containerID="d15ef001caec8a7c00633e7f071acb6486374af60e922b63a650b3791a01421c" Nov 03 00:17:32.631971 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:32.631856 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d15ef001caec8a7c00633e7f071acb6486374af60e922b63a650b3791a01421c"} err="failed to get container status \"d15ef001caec8a7c00633e7f071acb6486374af60e922b63a650b3791a01421c\": rpc error: code = NotFound desc = could not find container \"d15ef001caec8a7c00633e7f071acb6486374af60e922b63a650b3791a01421c\": container with ID starting with d15ef001caec8a7c00633e7f071acb6486374af60e922b63a650b3791a01421c not found: ID does not exist" Nov 03 00:17:32.780778 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:32.780748 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw"] Nov 03 00:17:32.845725 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:32.843327 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw"] Nov 03 00:17:33.148340 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:33.148264 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="365ba2a8-0d25-4d3e-b6b0-eb577be48fcd" path="/var/lib/kubelet/pods/365ba2a8-0d25-4d3e-b6b0-eb577be48fcd/volumes" Nov 03 00:17:35.016540 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:35.016495 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-ba1918-559cc4b5d-7mvqw" podUID="365ba2a8-0d25-4d3e-b6b0-eb577be48fcd" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.67:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:17:39.132500 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:39.132466 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4"] Nov 03 00:17:39.132952 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:39.132694 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" podUID="c6827543-3552-4c6c-bfed-203b1081cdc2" containerName="istio-proxy" containerID="cri-o://30b6ca2cfbc73992668ca258d859d464214df7aad294058eeafeb63c24278cb7" gracePeriod=30 Nov 03 00:17:39.883123 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:17:39.883084 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:17:44.396681 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:44.396617 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" Nov 03 00:17:44.537058 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:44.537021 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6827543-3552-4c6c-bfed-203b1081cdc2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6827543-3552-4c6c-bfed-203b1081cdc2-istio-envoy\") pod \"c6827543-3552-4c6c-bfed-203b1081cdc2\" (UID: \"c6827543-3552-4c6c-bfed-203b1081cdc2\") " Nov 03 00:17:44.537229 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:44.537106 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6827543-3552-4c6c-bfed-203b1081cdc2-kube-api-access-sb2vz\" (UniqueName: \"kubernetes.io/projected/c6827543-3552-4c6c-bfed-203b1081cdc2-kube-api-access-sb2vz\") pod \"c6827543-3552-4c6c-bfed-203b1081cdc2\" (UID: \"c6827543-3552-4c6c-bfed-203b1081cdc2\") " Nov 03 00:17:44.537229 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:44.537144 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6827543-3552-4c6c-bfed-203b1081cdc2-istio-token\" (UniqueName: \"kubernetes.io/projected/c6827543-3552-4c6c-bfed-203b1081cdc2-istio-token\") pod \"c6827543-3552-4c6c-bfed-203b1081cdc2\" (UID: \"c6827543-3552-4c6c-bfed-203b1081cdc2\") " Nov 03 00:17:44.537229 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:44.537166 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6827543-3552-4c6c-bfed-203b1081cdc2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6827543-3552-4c6c-bfed-203b1081cdc2-workload-socket\") pod \"c6827543-3552-4c6c-bfed-203b1081cdc2\" (UID: \"c6827543-3552-4c6c-bfed-203b1081cdc2\") " Nov 03 00:17:44.537229 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:44.537205 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6827543-3552-4c6c-bfed-203b1081cdc2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6827543-3552-4c6c-bfed-203b1081cdc2-credential-socket\") pod \"c6827543-3552-4c6c-bfed-203b1081cdc2\" (UID: \"c6827543-3552-4c6c-bfed-203b1081cdc2\") " Nov 03 00:17:44.537454 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:44.537242 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6827543-3552-4c6c-bfed-203b1081cdc2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6827543-3552-4c6c-bfed-203b1081cdc2-workload-certs\") pod \"c6827543-3552-4c6c-bfed-203b1081cdc2\" (UID: \"c6827543-3552-4c6c-bfed-203b1081cdc2\") " Nov 03 00:17:44.537454 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:44.537270 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6827543-3552-4c6c-bfed-203b1081cdc2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6827543-3552-4c6c-bfed-203b1081cdc2-istio-data\") pod \"c6827543-3552-4c6c-bfed-203b1081cdc2\" (UID: \"c6827543-3552-4c6c-bfed-203b1081cdc2\") " Nov 03 00:17:44.537454 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:44.537324 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6827543-3552-4c6c-bfed-203b1081cdc2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6827543-3552-4c6c-bfed-203b1081cdc2-istio-podinfo\") pod \"c6827543-3552-4c6c-bfed-203b1081cdc2\" (UID: \"c6827543-3552-4c6c-bfed-203b1081cdc2\") " Nov 03 00:17:44.537454 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:44.537388 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6827543-3552-4c6c-bfed-203b1081cdc2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6827543-3552-4c6c-bfed-203b1081cdc2-istiod-ca-cert\") pod \"c6827543-3552-4c6c-bfed-203b1081cdc2\" (UID: \"c6827543-3552-4c6c-bfed-203b1081cdc2\") " Nov 03 00:17:44.537685 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:44.537532 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6827543-3552-4c6c-bfed-203b1081cdc2-workload-certs" pod "c6827543-3552-4c6c-bfed-203b1081cdc2" (UID: "c6827543-3552-4c6c-bfed-203b1081cdc2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:44.537685 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:44.537539 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6827543-3552-4c6c-bfed-203b1081cdc2-credential-socket" pod "c6827543-3552-4c6c-bfed-203b1081cdc2" (UID: "c6827543-3552-4c6c-bfed-203b1081cdc2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:44.537807 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:44.537683 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6827543-3552-4c6c-bfed-203b1081cdc2-istio-data" pod "c6827543-3552-4c6c-bfed-203b1081cdc2" (UID: "c6827543-3552-4c6c-bfed-203b1081cdc2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:44.537807 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:44.537772 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6827543-3552-4c6c-bfed-203b1081cdc2-workload-socket" pod "c6827543-3552-4c6c-bfed-203b1081cdc2" (UID: "c6827543-3552-4c6c-bfed-203b1081cdc2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:44.537941 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:44.537865 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6827543-3552-4c6c-bfed-203b1081cdc2-istiod-ca-cert" pod "c6827543-3552-4c6c-bfed-203b1081cdc2" (UID: "c6827543-3552-4c6c-bfed-203b1081cdc2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:17:44.538043 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:44.538017 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6827543-3552-4c6c-bfed-203b1081cdc2-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:44.538201 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:44.538047 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6827543-3552-4c6c-bfed-203b1081cdc2-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:44.538201 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:44.538063 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6827543-3552-4c6c-bfed-203b1081cdc2-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:44.538201 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:44.538076 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6827543-3552-4c6c-bfed-203b1081cdc2-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:44.538201 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:44.538089 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6827543-3552-4c6c-bfed-203b1081cdc2-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:44.539495 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:44.539475 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6827543-3552-4c6c-bfed-203b1081cdc2-istio-token" pod "c6827543-3552-4c6c-bfed-203b1081cdc2" (UID: "c6827543-3552-4c6c-bfed-203b1081cdc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:17:44.539592 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:44.539578 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6827543-3552-4c6c-bfed-203b1081cdc2-istio-podinfo" pod "c6827543-3552-4c6c-bfed-203b1081cdc2" (UID: "c6827543-3552-4c6c-bfed-203b1081cdc2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:17:44.539736 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:44.539715 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6827543-3552-4c6c-bfed-203b1081cdc2-istio-envoy" pod "c6827543-3552-4c6c-bfed-203b1081cdc2" (UID: "c6827543-3552-4c6c-bfed-203b1081cdc2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:44.539818 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:44.539746 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6827543-3552-4c6c-bfed-203b1081cdc2-kube-api-access-sb2vz" pod "c6827543-3552-4c6c-bfed-203b1081cdc2" (UID: "c6827543-3552-4c6c-bfed-203b1081cdc2"). InnerVolumeSpecName "kube-api-access-sb2vz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:17:44.639705 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:44.639672 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6827543-3552-4c6c-bfed-203b1081cdc2-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:44.639705 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:44.639705 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb2vz\" (UniqueName: \"kubernetes.io/projected/c6827543-3552-4c6c-bfed-203b1081cdc2-kube-api-access-sb2vz\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:44.639928 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:44.639716 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6827543-3552-4c6c-bfed-203b1081cdc2-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:44.639928 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:44.639725 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6827543-3552-4c6c-bfed-203b1081cdc2-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:44.697777 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:44.697745 5076 generic.go:356] "Generic (PLEG): container finished" podID="c6827543-3552-4c6c-bfed-203b1081cdc2" containerID="30b6ca2cfbc73992668ca258d859d464214df7aad294058eeafeb63c24278cb7" exitCode=0 Nov 03 00:17:44.697978 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:44.697795 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" Nov 03 00:17:44.697978 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:44.697827 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" event={"ID":"c6827543-3552-4c6c-bfed-203b1081cdc2","Type":"ContainerDied","Data":"30b6ca2cfbc73992668ca258d859d464214df7aad294058eeafeb63c24278cb7"} Nov 03 00:17:44.697978 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:44.697925 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4" event={"ID":"c6827543-3552-4c6c-bfed-203b1081cdc2","Type":"ContainerDied","Data":"e46958704ce350372af5c360f0db3a5ef131560ea623d82415175f7f84d29974"} Nov 03 00:17:44.697978 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:44.697949 5076 scope.go:122] "RemoveContainer" containerID="30b6ca2cfbc73992668ca258d859d464214df7aad294058eeafeb63c24278cb7" Nov 03 00:17:44.708293 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:44.708277 5076 scope.go:122] "RemoveContainer" containerID="30b6ca2cfbc73992668ca258d859d464214df7aad294058eeafeb63c24278cb7" Nov 03 00:17:44.708543 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:17:44.708525 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30b6ca2cfbc73992668ca258d859d464214df7aad294058eeafeb63c24278cb7\": container with ID starting with 30b6ca2cfbc73992668ca258d859d464214df7aad294058eeafeb63c24278cb7 not found: ID does not exist" containerID="30b6ca2cfbc73992668ca258d859d464214df7aad294058eeafeb63c24278cb7" Nov 03 00:17:44.708590 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:44.708553 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30b6ca2cfbc73992668ca258d859d464214df7aad294058eeafeb63c24278cb7"} err="failed to get container status \"30b6ca2cfbc73992668ca258d859d464214df7aad294058eeafeb63c24278cb7\": rpc error: code = NotFound desc = could not find container \"30b6ca2cfbc73992668ca258d859d464214df7aad294058eeafeb63c24278cb7\": container with ID starting with 30b6ca2cfbc73992668ca258d859d464214df7aad294058eeafeb63c24278cb7 not found: ID does not exist" Nov 03 00:17:44.859565 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:44.859514 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4"] Nov 03 00:17:44.865218 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:44.865191 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2k4v4"] Nov 03 00:17:45.148066 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:45.147986 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6827543-3552-4c6c-bfed-203b1081cdc2" path="/var/lib/kubelet/pods/c6827543-3552-4c6c-bfed-203b1081cdc2/volumes" Nov 03 00:17:57.139175 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:57.139140 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz"] Nov 03 00:17:57.139635 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:57.139409 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz" podUID="e0ae7fef-665b-4ac5-9028-9214d1a9fe47" containerName="maas-api" containerID="cri-o://9a33710e99c68bbc052f578515aaf8227f6f81ce0cf33e0d69206ee9ceeca769" gracePeriod=30 Nov 03 00:17:57.425181 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:57.425150 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz" Nov 03 00:17:57.579664 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:57.579622 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0ae7fef-665b-4ac5-9028-9214d1a9fe47-kube-api-access-vlf49\" (UniqueName: \"kubernetes.io/projected/e0ae7fef-665b-4ac5-9028-9214d1a9fe47-kube-api-access-vlf49\") pod \"e0ae7fef-665b-4ac5-9028-9214d1a9fe47\" (UID: \"e0ae7fef-665b-4ac5-9028-9214d1a9fe47\") " Nov 03 00:17:57.579878 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:57.579681 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0ae7fef-665b-4ac5-9028-9214d1a9fe47-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0ae7fef-665b-4ac5-9028-9214d1a9fe47-openshift-service-ca\") pod \"e0ae7fef-665b-4ac5-9028-9214d1a9fe47\" (UID: \"e0ae7fef-665b-4ac5-9028-9214d1a9fe47\") " Nov 03 00:17:57.579878 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:57.579722 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0ae7fef-665b-4ac5-9028-9214d1a9fe47-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0ae7fef-665b-4ac5-9028-9214d1a9fe47-maas-api-tls\") pod \"e0ae7fef-665b-4ac5-9028-9214d1a9fe47\" (UID: \"e0ae7fef-665b-4ac5-9028-9214d1a9fe47\") " Nov 03 00:17:57.579878 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:57.579745 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0ae7fef-665b-4ac5-9028-9214d1a9fe47-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0ae7fef-665b-4ac5-9028-9214d1a9fe47-metrics-certs\") pod \"e0ae7fef-665b-4ac5-9028-9214d1a9fe47\" (UID: \"e0ae7fef-665b-4ac5-9028-9214d1a9fe47\") " Nov 03 00:17:57.580356 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:57.580166 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0ae7fef-665b-4ac5-9028-9214d1a9fe47-openshift-service-ca" pod "e0ae7fef-665b-4ac5-9028-9214d1a9fe47" (UID: "e0ae7fef-665b-4ac5-9028-9214d1a9fe47"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:17:57.582167 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:57.582142 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0ae7fef-665b-4ac5-9028-9214d1a9fe47-metrics-certs" pod "e0ae7fef-665b-4ac5-9028-9214d1a9fe47" (UID: "e0ae7fef-665b-4ac5-9028-9214d1a9fe47"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:17:57.582508 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:57.582480 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ae7fef-665b-4ac5-9028-9214d1a9fe47-kube-api-access-vlf49" pod "e0ae7fef-665b-4ac5-9028-9214d1a9fe47" (UID: "e0ae7fef-665b-4ac5-9028-9214d1a9fe47"). InnerVolumeSpecName "kube-api-access-vlf49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:17:57.582599 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:57.582533 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0ae7fef-665b-4ac5-9028-9214d1a9fe47-maas-api-tls" pod "e0ae7fef-665b-4ac5-9028-9214d1a9fe47" (UID: "e0ae7fef-665b-4ac5-9028-9214d1a9fe47"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:17:57.681033 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:57.680947 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlf49\" (UniqueName: \"kubernetes.io/projected/e0ae7fef-665b-4ac5-9028-9214d1a9fe47-kube-api-access-vlf49\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:57.681033 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:57.680978 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0ae7fef-665b-4ac5-9028-9214d1a9fe47-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:57.681033 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:57.680990 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0ae7fef-665b-4ac5-9028-9214d1a9fe47-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:57.681033 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:57.680998 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0ae7fef-665b-4ac5-9028-9214d1a9fe47-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:57.772637 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:57.772601 5076 generic.go:356] "Generic (PLEG): container finished" podID="e0ae7fef-665b-4ac5-9028-9214d1a9fe47" containerID="9a33710e99c68bbc052f578515aaf8227f6f81ce0cf33e0d69206ee9ceeca769" exitCode=0 Nov 03 00:17:57.772822 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:57.772659 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz" Nov 03 00:17:57.772822 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:57.772678 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz" event={"ID":"e0ae7fef-665b-4ac5-9028-9214d1a9fe47","Type":"ContainerDied","Data":"9a33710e99c68bbc052f578515aaf8227f6f81ce0cf33e0d69206ee9ceeca769"} Nov 03 00:17:57.772822 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:57.772730 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz" event={"ID":"e0ae7fef-665b-4ac5-9028-9214d1a9fe47","Type":"ContainerDied","Data":"b07473830ddf0c60f6252c35df81647c1e18f50273e68155a2702a38e492e775"} Nov 03 00:17:57.772822 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:57.772758 5076 scope.go:122] "RemoveContainer" containerID="9a33710e99c68bbc052f578515aaf8227f6f81ce0cf33e0d69206ee9ceeca769" Nov 03 00:17:57.783798 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:57.783779 5076 scope.go:122] "RemoveContainer" containerID="9a33710e99c68bbc052f578515aaf8227f6f81ce0cf33e0d69206ee9ceeca769" Nov 03 00:17:57.784113 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:17:57.784079 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a33710e99c68bbc052f578515aaf8227f6f81ce0cf33e0d69206ee9ceeca769\": container with ID starting with 9a33710e99c68bbc052f578515aaf8227f6f81ce0cf33e0d69206ee9ceeca769 not found: ID does not exist" containerID="9a33710e99c68bbc052f578515aaf8227f6f81ce0cf33e0d69206ee9ceeca769" Nov 03 00:17:57.784226 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:57.784123 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a33710e99c68bbc052f578515aaf8227f6f81ce0cf33e0d69206ee9ceeca769"} err="failed to get container status \"9a33710e99c68bbc052f578515aaf8227f6f81ce0cf33e0d69206ee9ceeca769\": rpc error: code = NotFound desc = could not find container \"9a33710e99c68bbc052f578515aaf8227f6f81ce0cf33e0d69206ee9ceeca769\": container with ID starting with 9a33710e99c68bbc052f578515aaf8227f6f81ce0cf33e0d69206ee9ceeca769 not found: ID does not exist" Nov 03 00:17:57.933823 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:57.933728 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz"] Nov 03 00:17:57.981308 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:57.981270 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz"] Nov 03 00:17:59.147217 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:17:59.147184 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0ae7fef-665b-4ac5-9028-9214d1a9fe47" path="/var/lib/kubelet/pods/e0ae7fef-665b-4ac5-9028-9214d1a9fe47/volumes" Nov 03 00:18:00.383129 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:00.383091 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ca60f-64456db76b-gf5qz" podUID="e0ae7fef-665b-4ac5-9028-9214d1a9fe47" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.168:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:18:09.695965 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:09.695929 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx"] Nov 03 00:18:09.884364 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:18:09.884318 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:18:24.717726 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:24.717687 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx" podUID="ffe70026-3a9a-49bb-bc08-e456e35a7c68" containerName="main" containerID="cri-o://b873d663442206882a2a1a2d2a872a69057ec5dcd3ca7f1ab0df46a64d0b13fa" gracePeriod=45 Nov 03 00:18:24.941807 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:24.941769 5076 generic.go:356] "Generic (PLEG): container finished" podID="ffe70026-3a9a-49bb-bc08-e456e35a7c68" containerID="b873d663442206882a2a1a2d2a872a69057ec5dcd3ca7f1ab0df46a64d0b13fa" exitCode=0 Nov 03 00:18:24.942018 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:24.941821 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx" event={"ID":"ffe70026-3a9a-49bb-bc08-e456e35a7c68","Type":"ContainerDied","Data":"b873d663442206882a2a1a2d2a872a69057ec5dcd3ca7f1ab0df46a64d0b13fa"} Nov 03 00:18:24.985776 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:24.985748 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx" Nov 03 00:18:25.060215 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:25.060180 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffe70026-3a9a-49bb-bc08-e456e35a7c68-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffe70026-3a9a-49bb-bc08-e456e35a7c68-tmp-dir\") pod \"ffe70026-3a9a-49bb-bc08-e456e35a7c68\" (UID: \"ffe70026-3a9a-49bb-bc08-e456e35a7c68\") " Nov 03 00:18:25.060394 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:25.060273 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffe70026-3a9a-49bb-bc08-e456e35a7c68-tls-certs\" (UniqueName: \"kubernetes.io/secret/ffe70026-3a9a-49bb-bc08-e456e35a7c68-tls-certs\") pod \"ffe70026-3a9a-49bb-bc08-e456e35a7c68\" (UID: \"ffe70026-3a9a-49bb-bc08-e456e35a7c68\") " Nov 03 00:18:25.060394 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:25.060340 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffe70026-3a9a-49bb-bc08-e456e35a7c68-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffe70026-3a9a-49bb-bc08-e456e35a7c68-model-cache\") pod \"ffe70026-3a9a-49bb-bc08-e456e35a7c68\" (UID: \"ffe70026-3a9a-49bb-bc08-e456e35a7c68\") " Nov 03 00:18:25.060394 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:25.060373 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffe70026-3a9a-49bb-bc08-e456e35a7c68-home\" (UniqueName: \"kubernetes.io/empty-dir/ffe70026-3a9a-49bb-bc08-e456e35a7c68-home\") pod \"ffe70026-3a9a-49bb-bc08-e456e35a7c68\" (UID: \"ffe70026-3a9a-49bb-bc08-e456e35a7c68\") " Nov 03 00:18:25.060556 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:25.060404 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffe70026-3a9a-49bb-bc08-e456e35a7c68-kube-api-access-s8r8x\" (UniqueName: \"kubernetes.io/projected/ffe70026-3a9a-49bb-bc08-e456e35a7c68-kube-api-access-s8r8x\") pod \"ffe70026-3a9a-49bb-bc08-e456e35a7c68\" (UID: \"ffe70026-3a9a-49bb-bc08-e456e35a7c68\") " Nov 03 00:18:25.060556 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:25.060428 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe70026-3a9a-49bb-bc08-e456e35a7c68-tmp-dir" pod "ffe70026-3a9a-49bb-bc08-e456e35a7c68" (UID: "ffe70026-3a9a-49bb-bc08-e456e35a7c68"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:18:25.060556 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:25.060473 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffe70026-3a9a-49bb-bc08-e456e35a7c68-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffe70026-3a9a-49bb-bc08-e456e35a7c68-dshm\") pod \"ffe70026-3a9a-49bb-bc08-e456e35a7c68\" (UID: \"ffe70026-3a9a-49bb-bc08-e456e35a7c68\") " Nov 03 00:18:25.060730 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:25.060646 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe70026-3a9a-49bb-bc08-e456e35a7c68-home" pod "ffe70026-3a9a-49bb-bc08-e456e35a7c68" (UID: "ffe70026-3a9a-49bb-bc08-e456e35a7c68"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:18:25.060730 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:25.060660 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe70026-3a9a-49bb-bc08-e456e35a7c68-model-cache" pod "ffe70026-3a9a-49bb-bc08-e456e35a7c68" (UID: "ffe70026-3a9a-49bb-bc08-e456e35a7c68"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:18:25.061169 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:25.061139 5076 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffe70026-3a9a-49bb-bc08-e456e35a7c68-tmp-dir\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:25.061262 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:25.061169 5076 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffe70026-3a9a-49bb-bc08-e456e35a7c68-model-cache\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:25.061262 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:25.061185 5076 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffe70026-3a9a-49bb-bc08-e456e35a7c68-home\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:25.062641 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:25.062617 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffe70026-3a9a-49bb-bc08-e456e35a7c68-tls-certs" pod "ffe70026-3a9a-49bb-bc08-e456e35a7c68" (UID: "ffe70026-3a9a-49bb-bc08-e456e35a7c68"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:18:25.063008 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:25.062987 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe70026-3a9a-49bb-bc08-e456e35a7c68-dshm" pod "ffe70026-3a9a-49bb-bc08-e456e35a7c68" (UID: "ffe70026-3a9a-49bb-bc08-e456e35a7c68"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:18:25.063087 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:25.063012 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffe70026-3a9a-49bb-bc08-e456e35a7c68-kube-api-access-s8r8x" pod "ffe70026-3a9a-49bb-bc08-e456e35a7c68" (UID: "ffe70026-3a9a-49bb-bc08-e456e35a7c68"). InnerVolumeSpecName "kube-api-access-s8r8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:18:25.162136 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:25.162108 5076 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffe70026-3a9a-49bb-bc08-e456e35a7c68-tls-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:25.162136 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:25.162136 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8r8x\" (UniqueName: \"kubernetes.io/projected/ffe70026-3a9a-49bb-bc08-e456e35a7c68-kube-api-access-s8r8x\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:25.162309 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:25.162147 5076 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffe70026-3a9a-49bb-bc08-e456e35a7c68-dshm\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:25.949599 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:25.949565 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx" Nov 03 00:18:25.949599 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:25.949589 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx" event={"ID":"ffe70026-3a9a-49bb-bc08-e456e35a7c68","Type":"ContainerDied","Data":"b12d3ca7c3d02eba1893823a59cd947c0208daa82440bab80adf571fa8920617"} Nov 03 00:18:25.950118 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:25.949644 5076 scope.go:122] "RemoveContainer" containerID="b873d663442206882a2a1a2d2a872a69057ec5dcd3ca7f1ab0df46a64d0b13fa" Nov 03 00:18:26.114944 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:26.114907 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx"] Nov 03 00:18:26.120715 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:26.120689 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-14b321/distinct-2-w3-14b321-kserve-6cb766ddc4-9qjmx"] Nov 03 00:18:27.148221 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:27.148187 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffe70026-3a9a-49bb-bc08-e456e35a7c68" path="/var/lib/kubelet/pods/ffe70026-3a9a-49bb-bc08-e456e35a7c68/volumes" Nov 03 00:18:39.883928 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:18:39.883873 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:18:42.419454 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:42.419373 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts"] Nov 03 00:18:42.420010 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:42.419572 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts" podUID="cf8cc281-8664-4196-8c74-d0b92d2ae815" containerName="maas-api" containerID="cri-o://f438a6be9a4859bc49f30bcac35be633d26f58276a142fcf23879a2b2d14f3c4" gracePeriod=30 Nov 03 00:18:42.679599 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:42.679535 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts" Nov 03 00:18:42.756516 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:42.756482 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf8cc281-8664-4196-8c74-d0b92d2ae815-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf8cc281-8664-4196-8c74-d0b92d2ae815-openshift-service-ca\") pod \"cf8cc281-8664-4196-8c74-d0b92d2ae815\" (UID: \"cf8cc281-8664-4196-8c74-d0b92d2ae815\") " Nov 03 00:18:42.756697 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:42.756527 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf8cc281-8664-4196-8c74-d0b92d2ae815-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf8cc281-8664-4196-8c74-d0b92d2ae815-maas-api-tls\") pod \"cf8cc281-8664-4196-8c74-d0b92d2ae815\" (UID: \"cf8cc281-8664-4196-8c74-d0b92d2ae815\") " Nov 03 00:18:42.756697 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:42.756560 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf8cc281-8664-4196-8c74-d0b92d2ae815-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf8cc281-8664-4196-8c74-d0b92d2ae815-metrics-certs\") pod \"cf8cc281-8664-4196-8c74-d0b92d2ae815\" (UID: \"cf8cc281-8664-4196-8c74-d0b92d2ae815\") " Nov 03 00:18:42.756697 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:42.756616 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf8cc281-8664-4196-8c74-d0b92d2ae815-kube-api-access-7fsqr\" (UniqueName: \"kubernetes.io/projected/cf8cc281-8664-4196-8c74-d0b92d2ae815-kube-api-access-7fsqr\") pod \"cf8cc281-8664-4196-8c74-d0b92d2ae815\" (UID: \"cf8cc281-8664-4196-8c74-d0b92d2ae815\") " Nov 03 00:18:42.756974 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:42.756947 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf8cc281-8664-4196-8c74-d0b92d2ae815-openshift-service-ca" pod "cf8cc281-8664-4196-8c74-d0b92d2ae815" (UID: "cf8cc281-8664-4196-8c74-d0b92d2ae815"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:18:42.758747 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:42.758724 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf8cc281-8664-4196-8c74-d0b92d2ae815-maas-api-tls" pod "cf8cc281-8664-4196-8c74-d0b92d2ae815" (UID: "cf8cc281-8664-4196-8c74-d0b92d2ae815"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:18:42.758861 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:42.758755 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf8cc281-8664-4196-8c74-d0b92d2ae815-metrics-certs" pod "cf8cc281-8664-4196-8c74-d0b92d2ae815" (UID: "cf8cc281-8664-4196-8c74-d0b92d2ae815"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:18:42.758861 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:42.758765 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf8cc281-8664-4196-8c74-d0b92d2ae815-kube-api-access-7fsqr" pod "cf8cc281-8664-4196-8c74-d0b92d2ae815" (UID: "cf8cc281-8664-4196-8c74-d0b92d2ae815"). InnerVolumeSpecName "kube-api-access-7fsqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:18:42.858314 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:42.858283 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf8cc281-8664-4196-8c74-d0b92d2ae815-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:42.858314 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:42.858311 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf8cc281-8664-4196-8c74-d0b92d2ae815-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:42.858314 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:42.858321 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf8cc281-8664-4196-8c74-d0b92d2ae815-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:42.858559 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:42.858329 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fsqr\" (UniqueName: \"kubernetes.io/projected/cf8cc281-8664-4196-8c74-d0b92d2ae815-kube-api-access-7fsqr\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:43.043262 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:43.043229 5076 generic.go:356] "Generic (PLEG): container finished" podID="cf8cc281-8664-4196-8c74-d0b92d2ae815" containerID="f438a6be9a4859bc49f30bcac35be633d26f58276a142fcf23879a2b2d14f3c4" exitCode=0 Nov 03 00:18:43.043451 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:43.043276 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts" Nov 03 00:18:43.043451 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:43.043313 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts" event={"ID":"cf8cc281-8664-4196-8c74-d0b92d2ae815","Type":"ContainerDied","Data":"f438a6be9a4859bc49f30bcac35be633d26f58276a142fcf23879a2b2d14f3c4"} Nov 03 00:18:43.043451 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:43.043367 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts" event={"ID":"cf8cc281-8664-4196-8c74-d0b92d2ae815","Type":"ContainerDied","Data":"7729e5dfd4703877cf67b7f8836ec6b59ada4c2ba85bed8ee4e5af40ea01a4c3"} Nov 03 00:18:43.043451 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:43.043395 5076 scope.go:122] "RemoveContainer" containerID="f438a6be9a4859bc49f30bcac35be633d26f58276a142fcf23879a2b2d14f3c4" Nov 03 00:18:43.054102 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:43.054083 5076 scope.go:122] "RemoveContainer" containerID="f438a6be9a4859bc49f30bcac35be633d26f58276a142fcf23879a2b2d14f3c4" Nov 03 00:18:43.054367 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:18:43.054348 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f438a6be9a4859bc49f30bcac35be633d26f58276a142fcf23879a2b2d14f3c4\": container with ID starting with f438a6be9a4859bc49f30bcac35be633d26f58276a142fcf23879a2b2d14f3c4 not found: ID does not exist" containerID="f438a6be9a4859bc49f30bcac35be633d26f58276a142fcf23879a2b2d14f3c4" Nov 03 00:18:43.054452 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:43.054378 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f438a6be9a4859bc49f30bcac35be633d26f58276a142fcf23879a2b2d14f3c4"} err="failed to get container status \"f438a6be9a4859bc49f30bcac35be633d26f58276a142fcf23879a2b2d14f3c4\": rpc error: code = NotFound desc = could not find container \"f438a6be9a4859bc49f30bcac35be633d26f58276a142fcf23879a2b2d14f3c4\": container with ID starting with f438a6be9a4859bc49f30bcac35be633d26f58276a142fcf23879a2b2d14f3c4 not found: ID does not exist" Nov 03 00:18:43.208355 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:43.208321 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts"] Nov 03 00:18:43.211214 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:43.211190 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-14b321-97b4dcf7d-zlfts"] Nov 03 00:18:45.147778 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:45.147741 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf8cc281-8664-4196-8c74-d0b92d2ae815" path="/var/lib/kubelet/pods/cf8cc281-8664-4196-8c74-d0b92d2ae815/volumes" Nov 03 00:18:56.771920 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:56.770416 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n"] Nov 03 00:18:56.771920 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:18:56.770765 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" podUID="f6b0b2ba-6e13-4d79-b2b1-243f80b5703c" containerName="istio-proxy" containerID="cri-o://fd114a8aad01ca77a63c98e495907d5bf45fbbaa37c4200ee7dae82f1572c54e" gracePeriod=30 Nov 03 00:19:00.753604 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:00.753568 5076 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:19:00.754040 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:00.753633 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" podUID="f6b0b2ba-6e13-4d79-b2b1-243f80b5703c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:19:02.044585 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.044511 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" Nov 03 00:19:02.160006 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.159968 5076 generic.go:356] "Generic (PLEG): container finished" podID="f6b0b2ba-6e13-4d79-b2b1-243f80b5703c" containerID="fd114a8aad01ca77a63c98e495907d5bf45fbbaa37c4200ee7dae82f1572c54e" exitCode=0 Nov 03 00:19:02.160006 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.160010 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" event={"ID":"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c","Type":"ContainerDied","Data":"fd114a8aad01ca77a63c98e495907d5bf45fbbaa37c4200ee7dae82f1572c54e"} Nov 03 00:19:02.160230 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.160043 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" Nov 03 00:19:02.160230 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.160083 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n" event={"ID":"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c","Type":"ContainerDied","Data":"e18c50700f0191f3a1140f770e2479e16fc6a40a85e70681bbd4e4424ba6330a"} Nov 03 00:19:02.160230 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.160106 5076 scope.go:122] "RemoveContainer" containerID="fd114a8aad01ca77a63c98e495907d5bf45fbbaa37c4200ee7dae82f1572c54e" Nov 03 00:19:02.160513 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.160497 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-credential-socket\") pod \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\" (UID: \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\") " Nov 03 00:19:02.160561 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.160530 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-workload-socket\") pod \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\" (UID: \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\") " Nov 03 00:19:02.160561 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.160549 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-workload-certs\") pod \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\" (UID: \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\") " Nov 03 00:19:02.160652 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.160600 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-istio-data\") pod \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\" (UID: \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\") " Nov 03 00:19:02.160706 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.160659 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-istio-token\" (UniqueName: \"kubernetes.io/projected/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-istio-token\") pod \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\" (UID: \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\") " Nov 03 00:19:02.160706 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.160696 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-istio-envoy\") pod \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\" (UID: \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\") " Nov 03 00:19:02.160811 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.160751 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-kube-api-access-cckfr\" (UniqueName: \"kubernetes.io/projected/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-kube-api-access-cckfr\") pod \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\" (UID: \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\") " Nov 03 00:19:02.160811 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.160780 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-istiod-ca-cert\") pod \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\" (UID: \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\") " Nov 03 00:19:02.160811 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.160783 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-credential-socket" pod "f6b0b2ba-6e13-4d79-b2b1-243f80b5703c" (UID: "f6b0b2ba-6e13-4d79-b2b1-243f80b5703c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:19:02.161015 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.160820 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-istio-podinfo\") pod \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\" (UID: \"f6b0b2ba-6e13-4d79-b2b1-243f80b5703c\") " Nov 03 00:19:02.161015 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.160864 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-workload-socket" pod "f6b0b2ba-6e13-4d79-b2b1-243f80b5703c" (UID: "f6b0b2ba-6e13-4d79-b2b1-243f80b5703c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:19:02.161015 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.160972 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-istio-data" pod "f6b0b2ba-6e13-4d79-b2b1-243f80b5703c" (UID: "f6b0b2ba-6e13-4d79-b2b1-243f80b5703c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:19:02.161015 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.160987 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-workload-certs" pod "f6b0b2ba-6e13-4d79-b2b1-243f80b5703c" (UID: "f6b0b2ba-6e13-4d79-b2b1-243f80b5703c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:19:02.161380 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.161271 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:19:02.161380 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.161293 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:19:02.161380 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.161303 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:19:02.161380 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.161315 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:19:02.161685 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.161660 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-istiod-ca-cert" pod "f6b0b2ba-6e13-4d79-b2b1-243f80b5703c" (UID: "f6b0b2ba-6e13-4d79-b2b1-243f80b5703c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:19:02.163182 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.163158 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-istio-envoy" pod "f6b0b2ba-6e13-4d79-b2b1-243f80b5703c" (UID: "f6b0b2ba-6e13-4d79-b2b1-243f80b5703c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:19:02.163284 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.163187 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-istio-podinfo" pod "f6b0b2ba-6e13-4d79-b2b1-243f80b5703c" (UID: "f6b0b2ba-6e13-4d79-b2b1-243f80b5703c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:19:02.163284 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.163194 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-kube-api-access-cckfr" pod "f6b0b2ba-6e13-4d79-b2b1-243f80b5703c" (UID: "f6b0b2ba-6e13-4d79-b2b1-243f80b5703c"). InnerVolumeSpecName "kube-api-access-cckfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:19:02.163400 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.163381 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-istio-token" pod "f6b0b2ba-6e13-4d79-b2b1-243f80b5703c" (UID: "f6b0b2ba-6e13-4d79-b2b1-243f80b5703c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:19:02.185146 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.185124 5076 scope.go:122] "RemoveContainer" containerID="e1f08f0a7fc6972d298961353624a3625a1ae5eeaea913dabd22615b1b391dd9" Nov 03 00:19:02.194381 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.194354 5076 scope.go:122] "RemoveContainer" containerID="fd114a8aad01ca77a63c98e495907d5bf45fbbaa37c4200ee7dae82f1572c54e" Nov 03 00:19:02.194680 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:19:02.194658 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd114a8aad01ca77a63c98e495907d5bf45fbbaa37c4200ee7dae82f1572c54e\": container with ID starting with fd114a8aad01ca77a63c98e495907d5bf45fbbaa37c4200ee7dae82f1572c54e not found: ID does not exist" containerID="fd114a8aad01ca77a63c98e495907d5bf45fbbaa37c4200ee7dae82f1572c54e" Nov 03 00:19:02.194748 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.194689 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd114a8aad01ca77a63c98e495907d5bf45fbbaa37c4200ee7dae82f1572c54e"} err="failed to get container status \"fd114a8aad01ca77a63c98e495907d5bf45fbbaa37c4200ee7dae82f1572c54e\": rpc error: code = NotFound desc = could not find container \"fd114a8aad01ca77a63c98e495907d5bf45fbbaa37c4200ee7dae82f1572c54e\": container with ID starting with fd114a8aad01ca77a63c98e495907d5bf45fbbaa37c4200ee7dae82f1572c54e not found: ID does not exist" Nov 03 00:19:02.194748 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.194709 5076 scope.go:122] "RemoveContainer" containerID="e1f08f0a7fc6972d298961353624a3625a1ae5eeaea913dabd22615b1b391dd9" Nov 03 00:19:02.195069 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:19:02.195050 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1f08f0a7fc6972d298961353624a3625a1ae5eeaea913dabd22615b1b391dd9\": container with ID starting with e1f08f0a7fc6972d298961353624a3625a1ae5eeaea913dabd22615b1b391dd9 not found: ID does not exist" containerID="e1f08f0a7fc6972d298961353624a3625a1ae5eeaea913dabd22615b1b391dd9" Nov 03 00:19:02.195131 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.195078 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1f08f0a7fc6972d298961353624a3625a1ae5eeaea913dabd22615b1b391dd9"} err="failed to get container status \"e1f08f0a7fc6972d298961353624a3625a1ae5eeaea913dabd22615b1b391dd9\": rpc error: code = NotFound desc = could not find container \"e1f08f0a7fc6972d298961353624a3625a1ae5eeaea913dabd22615b1b391dd9\": container with ID starting with e1f08f0a7fc6972d298961353624a3625a1ae5eeaea913dabd22615b1b391dd9 not found: ID does not exist" Nov 03 00:19:02.262116 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.262079 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:19:02.262116 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.262107 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:19:02.262116 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.262116 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:19:02.262116 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.262124 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:19:02.262454 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.262134 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cckfr\" (UniqueName: \"kubernetes.io/projected/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c-kube-api-access-cckfr\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:19:02.578918 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.578867 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n"] Nov 03 00:19:02.582798 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:02.582768 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-bv68n"] Nov 03 00:19:03.147634 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:03.147598 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6b0b2ba-6e13-4d79-b2b1-243f80b5703c" path="/var/lib/kubelet/pods/f6b0b2ba-6e13-4d79-b2b1-243f80b5703c/volumes" Nov 03 00:19:09.884115 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:19:09.884071 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:19:13.227655 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:13.227620 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt"] Nov 03 00:19:13.228196 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:13.227818 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt" podUID="79ab1c7e-f257-492f-ac2f-dfa72b75d711" containerName="maas-api" containerID="cri-o://0dd4d4e9542132b2cd759f9b529cb21497c00c9436b9e8e223d384fd4c524af1" gracePeriod=30 Nov 03 00:19:13.501207 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:13.501179 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt" Nov 03 00:19:13.677653 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:13.677618 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79ab1c7e-f257-492f-ac2f-dfa72b75d711-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ab1c7e-f257-492f-ac2f-dfa72b75d711-openshift-service-ca\") pod \"79ab1c7e-f257-492f-ac2f-dfa72b75d711\" (UID: \"79ab1c7e-f257-492f-ac2f-dfa72b75d711\") " Nov 03 00:19:13.677847 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:13.677698 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79ab1c7e-f257-492f-ac2f-dfa72b75d711-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79ab1c7e-f257-492f-ac2f-dfa72b75d711-maas-api-tls\") pod \"79ab1c7e-f257-492f-ac2f-dfa72b75d711\" (UID: \"79ab1c7e-f257-492f-ac2f-dfa72b75d711\") " Nov 03 00:19:13.677847 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:13.677729 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79ab1c7e-f257-492f-ac2f-dfa72b75d711-kube-api-access-k8ctd\" (UniqueName: \"kubernetes.io/projected/79ab1c7e-f257-492f-ac2f-dfa72b75d711-kube-api-access-k8ctd\") pod \"79ab1c7e-f257-492f-ac2f-dfa72b75d711\" (UID: \"79ab1c7e-f257-492f-ac2f-dfa72b75d711\") " Nov 03 00:19:13.677847 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:13.677777 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79ab1c7e-f257-492f-ac2f-dfa72b75d711-metrics-certs\" (UniqueName: \"kubernetes.io/secret/79ab1c7e-f257-492f-ac2f-dfa72b75d711-metrics-certs\") pod \"79ab1c7e-f257-492f-ac2f-dfa72b75d711\" (UID: \"79ab1c7e-f257-492f-ac2f-dfa72b75d711\") " Nov 03 00:19:13.678143 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:13.678078 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79ab1c7e-f257-492f-ac2f-dfa72b75d711-openshift-service-ca" pod "79ab1c7e-f257-492f-ac2f-dfa72b75d711" (UID: "79ab1c7e-f257-492f-ac2f-dfa72b75d711"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:19:13.678303 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:13.678271 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ab1c7e-f257-492f-ac2f-dfa72b75d711-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:19:13.679975 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:13.679950 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ab1c7e-f257-492f-ac2f-dfa72b75d711-kube-api-access-k8ctd" pod "79ab1c7e-f257-492f-ac2f-dfa72b75d711" (UID: "79ab1c7e-f257-492f-ac2f-dfa72b75d711"). InnerVolumeSpecName "kube-api-access-k8ctd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:19:13.679975 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:13.679965 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79ab1c7e-f257-492f-ac2f-dfa72b75d711-metrics-certs" pod "79ab1c7e-f257-492f-ac2f-dfa72b75d711" (UID: "79ab1c7e-f257-492f-ac2f-dfa72b75d711"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:19:13.680135 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:13.679979 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79ab1c7e-f257-492f-ac2f-dfa72b75d711-maas-api-tls" pod "79ab1c7e-f257-492f-ac2f-dfa72b75d711" (UID: "79ab1c7e-f257-492f-ac2f-dfa72b75d711"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:19:13.778975 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:13.778836 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79ab1c7e-f257-492f-ac2f-dfa72b75d711-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:19:13.778975 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:13.778866 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79ab1c7e-f257-492f-ac2f-dfa72b75d711-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:19:13.778975 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:13.778877 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8ctd\" (UniqueName: \"kubernetes.io/projected/79ab1c7e-f257-492f-ac2f-dfa72b75d711-kube-api-access-k8ctd\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:19:14.221633 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:14.221602 5076 generic.go:356] "Generic (PLEG): container finished" podID="79ab1c7e-f257-492f-ac2f-dfa72b75d711" containerID="0dd4d4e9542132b2cd759f9b529cb21497c00c9436b9e8e223d384fd4c524af1" exitCode=0 Nov 03 00:19:14.221819 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:14.221654 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt" event={"ID":"79ab1c7e-f257-492f-ac2f-dfa72b75d711","Type":"ContainerDied","Data":"0dd4d4e9542132b2cd759f9b529cb21497c00c9436b9e8e223d384fd4c524af1"} Nov 03 00:19:14.221819 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:14.221673 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt" Nov 03 00:19:14.221819 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:14.221696 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt" event={"ID":"79ab1c7e-f257-492f-ac2f-dfa72b75d711","Type":"ContainerDied","Data":"2e04c4fb316228324ed5f44de38d6f95f0e4650a22c5fae1adc4279ea47362f1"} Nov 03 00:19:14.221819 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:14.221717 5076 scope.go:122] "RemoveContainer" containerID="0dd4d4e9542132b2cd759f9b529cb21497c00c9436b9e8e223d384fd4c524af1" Nov 03 00:19:14.231571 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:14.231386 5076 scope.go:122] "RemoveContainer" containerID="0dd4d4e9542132b2cd759f9b529cb21497c00c9436b9e8e223d384fd4c524af1" Nov 03 00:19:14.231833 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:19:14.231684 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dd4d4e9542132b2cd759f9b529cb21497c00c9436b9e8e223d384fd4c524af1\": container with ID starting with 0dd4d4e9542132b2cd759f9b529cb21497c00c9436b9e8e223d384fd4c524af1 not found: ID does not exist" containerID="0dd4d4e9542132b2cd759f9b529cb21497c00c9436b9e8e223d384fd4c524af1" Nov 03 00:19:14.231833 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:14.231712 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dd4d4e9542132b2cd759f9b529cb21497c00c9436b9e8e223d384fd4c524af1"} err="failed to get container status \"0dd4d4e9542132b2cd759f9b529cb21497c00c9436b9e8e223d384fd4c524af1\": rpc error: code = NotFound desc = could not find container \"0dd4d4e9542132b2cd759f9b529cb21497c00c9436b9e8e223d384fd4c524af1\": container with ID starting with 0dd4d4e9542132b2cd759f9b529cb21497c00c9436b9e8e223d384fd4c524af1 not found: ID does not exist" Nov 03 00:19:14.384344 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:14.384302 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt"] Nov 03 00:19:14.389188 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:14.389160 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b4d53e-74c6f68554-2fvnt"] Nov 03 00:19:15.147216 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:15.147184 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79ab1c7e-f257-492f-ac2f-dfa72b75d711" path="/var/lib/kubelet/pods/79ab1c7e-f257-492f-ac2f-dfa72b75d711/volumes" Nov 03 00:19:19.799810 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:19.799772 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw"] Nov 03 00:19:19.800284 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:19.799993 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" podUID="ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2" containerName="istio-proxy" containerID="cri-o://a5dc776994907dbfa0df23116eba8ee961284b56dba108f127bfc4dbf0664655" gracePeriod=30 Nov 03 00:19:25.051197 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.051166 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" Nov 03 00:19:25.088878 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.088286 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-workload-socket\") pod \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\" (UID: \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\") " Nov 03 00:19:25.088878 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.088360 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-istio-data\") pod \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\" (UID: \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\") " Nov 03 00:19:25.088878 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.088393 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-workload-certs\") pod \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\" (UID: \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\") " Nov 03 00:19:25.088878 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.088438 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-kube-api-access-9cg4t\" (UniqueName: \"kubernetes.io/projected/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-kube-api-access-9cg4t\") pod \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\" (UID: \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\") " Nov 03 00:19:25.088878 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.088492 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-workload-socket" pod "ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2" (UID: "ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:19:25.088878 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.088771 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:19:25.089458 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.089020 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-istio-data" pod "ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2" (UID: "ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:19:25.089458 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.089035 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-workload-certs" pod "ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2" (UID: "ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:19:25.091481 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.091452 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-kube-api-access-9cg4t" pod "ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2" (UID: "ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2"). InnerVolumeSpecName "kube-api-access-9cg4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:19:25.189311 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.189276 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-credential-socket\") pod \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\" (UID: \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\") " Nov 03 00:19:25.189473 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.189444 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-istio-token\" (UniqueName: \"kubernetes.io/projected/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-istio-token\") pod \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\" (UID: \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\") " Nov 03 00:19:25.189541 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.189478 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-istiod-ca-cert\") pod \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\" (UID: \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\") " Nov 03 00:19:25.189541 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.189497 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-istio-envoy\") pod \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\" (UID: \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\") " Nov 03 00:19:25.189541 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.189505 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-credential-socket" pod "ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2" (UID: "ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:19:25.189701 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.189625 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-istio-podinfo\") pod \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\" (UID: \"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2\") " Nov 03 00:19:25.189788 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.189770 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-istiod-ca-cert" pod "ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2" (UID: "ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:19:25.190434 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.190410 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:19:25.190434 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.190436 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:19:25.190646 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.190447 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cg4t\" (UniqueName: \"kubernetes.io/projected/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-kube-api-access-9cg4t\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:19:25.190646 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.190458 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:19:25.190646 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.190467 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:19:25.191683 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.191656 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-istio-token" pod "ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2" (UID: "ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:19:25.191789 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.191686 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-istio-podinfo" pod "ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2" (UID: "ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:19:25.191789 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.191685 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-istio-envoy" pod "ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2" (UID: "ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:19:25.280789 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.280755 5076 generic.go:356] "Generic (PLEG): container finished" podID="ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2" containerID="a5dc776994907dbfa0df23116eba8ee961284b56dba108f127bfc4dbf0664655" exitCode=0 Nov 03 00:19:25.281024 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.280815 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" Nov 03 00:19:25.281024 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.280840 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" event={"ID":"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2","Type":"ContainerDied","Data":"a5dc776994907dbfa0df23116eba8ee961284b56dba108f127bfc4dbf0664655"} Nov 03 00:19:25.281024 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.280936 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw" event={"ID":"ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2","Type":"ContainerDied","Data":"a56b500f3b8cbca10f42c50090faeb6d4308bcc6bb3cc43d8da99f9dacfbd67a"} Nov 03 00:19:25.281024 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.280970 5076 scope.go:122] "RemoveContainer" containerID="a5dc776994907dbfa0df23116eba8ee961284b56dba108f127bfc4dbf0664655" Nov 03 00:19:25.290503 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.290416 5076 scope.go:122] "RemoveContainer" containerID="574dbc24446700e0bb5156ce4e1418b3041c05c7110bd1214dcaf87b8e76a31e" Nov 03 00:19:25.290933 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.290917 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:19:25.291002 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.290936 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:19:25.291002 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.290946 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:19:25.302384 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.302360 5076 scope.go:122] "RemoveContainer" containerID="a5dc776994907dbfa0df23116eba8ee961284b56dba108f127bfc4dbf0664655" Nov 03 00:19:25.302633 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:19:25.302611 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5dc776994907dbfa0df23116eba8ee961284b56dba108f127bfc4dbf0664655\": container with ID starting with a5dc776994907dbfa0df23116eba8ee961284b56dba108f127bfc4dbf0664655 not found: ID does not exist" containerID="a5dc776994907dbfa0df23116eba8ee961284b56dba108f127bfc4dbf0664655" Nov 03 00:19:25.302699 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.302642 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5dc776994907dbfa0df23116eba8ee961284b56dba108f127bfc4dbf0664655"} err="failed to get container status \"a5dc776994907dbfa0df23116eba8ee961284b56dba108f127bfc4dbf0664655\": rpc error: code = NotFound desc = could not find container \"a5dc776994907dbfa0df23116eba8ee961284b56dba108f127bfc4dbf0664655\": container with ID starting with a5dc776994907dbfa0df23116eba8ee961284b56dba108f127bfc4dbf0664655 not found: ID does not exist" Nov 03 00:19:25.302699 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.302668 5076 scope.go:122] "RemoveContainer" containerID="574dbc24446700e0bb5156ce4e1418b3041c05c7110bd1214dcaf87b8e76a31e" Nov 03 00:19:25.302941 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:19:25.302920 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"574dbc24446700e0bb5156ce4e1418b3041c05c7110bd1214dcaf87b8e76a31e\": container with ID starting with 574dbc24446700e0bb5156ce4e1418b3041c05c7110bd1214dcaf87b8e76a31e not found: ID does not exist" containerID="574dbc24446700e0bb5156ce4e1418b3041c05c7110bd1214dcaf87b8e76a31e" Nov 03 00:19:25.303009 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.302948 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"574dbc24446700e0bb5156ce4e1418b3041c05c7110bd1214dcaf87b8e76a31e"} err="failed to get container status \"574dbc24446700e0bb5156ce4e1418b3041c05c7110bd1214dcaf87b8e76a31e\": rpc error: code = NotFound desc = could not find container \"574dbc24446700e0bb5156ce4e1418b3041c05c7110bd1214dcaf87b8e76a31e\": container with ID starting with 574dbc24446700e0bb5156ce4e1418b3041c05c7110bd1214dcaf87b8e76a31e not found: ID does not exist" Nov 03 00:19:25.440857 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.440819 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw"] Nov 03 00:19:25.450396 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:25.450357 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-b4d53e-openshift-default-5f589d66fc-585kw"] Nov 03 00:19:27.146860 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:27.146822 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2" path="/var/lib/kubelet/pods/ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2/volumes" Nov 03 00:19:39.883264 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:19:39.883220 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:19:50.113444 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:50.113401 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j"] Nov 03 00:19:50.113996 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:50.113681 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" podUID="6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e" containerName="istio-proxy" containerID="cri-o://3926e367f244454e6746785c2f8fa3b7c4685c5d19139ac881082d875b310d7d" gracePeriod=30 Nov 03 00:19:55.364342 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.364316 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" Nov 03 00:19:55.439088 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.439059 5076 generic.go:356] "Generic (PLEG): container finished" podID="6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e" containerID="3926e367f244454e6746785c2f8fa3b7c4685c5d19139ac881082d875b310d7d" exitCode=0 Nov 03 00:19:55.439254 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.439107 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" event={"ID":"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e","Type":"ContainerDied","Data":"3926e367f244454e6746785c2f8fa3b7c4685c5d19139ac881082d875b310d7d"} Nov 03 00:19:55.439254 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.439152 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" event={"ID":"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e","Type":"ContainerDied","Data":"1b2da61166ae4790ff50115a5c1b0fa9a406a0966093db1b967503b33f59824e"} Nov 03 00:19:55.439254 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.439174 5076 scope.go:122] "RemoveContainer" containerID="3926e367f244454e6746785c2f8fa3b7c4685c5d19139ac881082d875b310d7d" Nov 03 00:19:55.439254 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.439111 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j" Nov 03 00:19:55.448278 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.448256 5076 scope.go:122] "RemoveContainer" containerID="ebc686f9abf6989bad70a068d8bc3eade1b424a6ee79d59586fb29d8850cb6df" Nov 03 00:19:55.456380 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.456363 5076 scope.go:122] "RemoveContainer" containerID="3926e367f244454e6746785c2f8fa3b7c4685c5d19139ac881082d875b310d7d" Nov 03 00:19:55.456671 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:19:55.456645 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3926e367f244454e6746785c2f8fa3b7c4685c5d19139ac881082d875b310d7d\": container with ID starting with 3926e367f244454e6746785c2f8fa3b7c4685c5d19139ac881082d875b310d7d not found: ID does not exist" containerID="3926e367f244454e6746785c2f8fa3b7c4685c5d19139ac881082d875b310d7d" Nov 03 00:19:55.456731 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.456682 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3926e367f244454e6746785c2f8fa3b7c4685c5d19139ac881082d875b310d7d"} err="failed to get container status \"3926e367f244454e6746785c2f8fa3b7c4685c5d19139ac881082d875b310d7d\": rpc error: code = NotFound desc = could not find container \"3926e367f244454e6746785c2f8fa3b7c4685c5d19139ac881082d875b310d7d\": container with ID starting with 3926e367f244454e6746785c2f8fa3b7c4685c5d19139ac881082d875b310d7d not found: ID does not exist" Nov 03 00:19:55.456731 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.456700 5076 scope.go:122] "RemoveContainer" containerID="ebc686f9abf6989bad70a068d8bc3eade1b424a6ee79d59586fb29d8850cb6df" Nov 03 00:19:55.456960 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:19:55.456943 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebc686f9abf6989bad70a068d8bc3eade1b424a6ee79d59586fb29d8850cb6df\": container with ID starting with ebc686f9abf6989bad70a068d8bc3eade1b424a6ee79d59586fb29d8850cb6df not found: ID does not exist" containerID="ebc686f9abf6989bad70a068d8bc3eade1b424a6ee79d59586fb29d8850cb6df" Nov 03 00:19:55.457008 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.456967 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebc686f9abf6989bad70a068d8bc3eade1b424a6ee79d59586fb29d8850cb6df"} err="failed to get container status \"ebc686f9abf6989bad70a068d8bc3eade1b424a6ee79d59586fb29d8850cb6df\": rpc error: code = NotFound desc = could not find container \"ebc686f9abf6989bad70a068d8bc3eade1b424a6ee79d59586fb29d8850cb6df\": container with ID starting with ebc686f9abf6989bad70a068d8bc3eade1b424a6ee79d59586fb29d8850cb6df not found: ID does not exist" Nov 03 00:19:55.501876 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.501847 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-istio-podinfo\") pod \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\" (UID: \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\") " Nov 03 00:19:55.501991 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.501883 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-istio-token\" (UniqueName: \"kubernetes.io/projected/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-istio-token\") pod \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\" (UID: \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\") " Nov 03 00:19:55.501991 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.501963 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-kube-api-access-zp7jt\" (UniqueName: \"kubernetes.io/projected/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-kube-api-access-zp7jt\") pod \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\" (UID: \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\") " Nov 03 00:19:55.502105 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.501987 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-workload-certs\") pod \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\" (UID: \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\") " Nov 03 00:19:55.502105 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.502038 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-istiod-ca-cert\") pod \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\" (UID: \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\") " Nov 03 00:19:55.502105 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.502077 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-workload-socket\") pod \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\" (UID: \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\") " Nov 03 00:19:55.502279 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.502152 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-credential-socket\") pod \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\" (UID: \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\") " Nov 03 00:19:55.502279 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.502193 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-istio-data\") pod \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\" (UID: \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\") " Nov 03 00:19:55.502396 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.502283 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-istio-envoy\") pod \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\" (UID: \"6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e\") " Nov 03 00:19:55.502625 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.502512 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-istiod-ca-cert" pod "6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e" (UID: "6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:19:55.502625 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.502505 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-workload-socket" pod "6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e" (UID: "6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:19:55.502625 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.502546 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-workload-certs" pod "6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e" (UID: "6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:19:55.502625 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.502573 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-istio-data" pod "6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e" (UID: "6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:19:55.502864 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.502740 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-credential-socket" pod "6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e" (UID: "6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:19:55.502864 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.502813 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:19:55.502864 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.502835 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:19:55.502864 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.502851 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:19:55.502864 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.502866 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:19:55.504179 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.504154 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-istio-podinfo" pod "6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e" (UID: "6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:19:55.504337 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.504199 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-istio-token" pod "6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e" (UID: "6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:19:55.504337 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.504250 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-kube-api-access-zp7jt" pod "6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e" (UID: "6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e"). InnerVolumeSpecName "kube-api-access-zp7jt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:19:55.504476 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.504448 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-istio-envoy" pod "6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e" (UID: "6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:19:55.520995 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.520970 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7"] Nov 03 00:19:55.572680 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.572647 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv"] Nov 03 00:19:55.603978 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.603945 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp7jt\" (UniqueName: \"kubernetes.io/projected/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-kube-api-access-zp7jt\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:19:55.603978 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.603971 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:19:55.603978 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.603983 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:19:55.604196 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.603993 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:19:55.604196 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.604003 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:19:55.945085 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.945041 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j"] Nov 03 00:19:55.953766 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:55.953739 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a51378-openshift-default-846fb67b55-v4f2j"] Nov 03 00:19:57.147167 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:19:57.147137 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e" path="/var/lib/kubelet/pods/6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e/volumes" Nov 03 00:20:09.885594 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:20:09.885554 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:20:10.540687 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:10.540646 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" podUID="cf2b01c6-8001-49e3-908a-38cffb88edfa" containerName="main" containerID="cri-o://161090faabb34d26de936b2a96fe2e49a0fbb1cf60a50ca1b8442b78ec1ae00a" gracePeriod=45 Nov 03 00:20:10.592712 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:10.592671 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" podUID="c5cd8831-584f-47bb-9580-4d41deab29dc" containerName="main" containerID="cri-o://cf6208e292a38fc8dda1168ff8fe0f6913b3ffd612d5ba88a732ae89136a2ad8" gracePeriod=45 Nov 03 00:20:10.859860 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:10.859835 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" Nov 03 00:20:10.885646 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:10.885622 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" Nov 03 00:20:10.952486 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:10.952453 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf2b01c6-8001-49e3-908a-38cffb88edfa-kube-api-access-pwdsg\" (UniqueName: \"kubernetes.io/projected/cf2b01c6-8001-49e3-908a-38cffb88edfa-kube-api-access-pwdsg\") pod \"cf2b01c6-8001-49e3-908a-38cffb88edfa\" (UID: \"cf2b01c6-8001-49e3-908a-38cffb88edfa\") " Nov 03 00:20:10.952673 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:10.952493 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf2b01c6-8001-49e3-908a-38cffb88edfa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf2b01c6-8001-49e3-908a-38cffb88edfa-model-cache\") pod \"cf2b01c6-8001-49e3-908a-38cffb88edfa\" (UID: \"cf2b01c6-8001-49e3-908a-38cffb88edfa\") " Nov 03 00:20:10.952673 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:10.952535 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5cd8831-584f-47bb-9580-4d41deab29dc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5cd8831-584f-47bb-9580-4d41deab29dc-dshm\") pod \"c5cd8831-584f-47bb-9580-4d41deab29dc\" (UID: \"c5cd8831-584f-47bb-9580-4d41deab29dc\") " Nov 03 00:20:10.952673 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:10.952556 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf2b01c6-8001-49e3-908a-38cffb88edfa-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf2b01c6-8001-49e3-908a-38cffb88edfa-tls-certs\") pod \"cf2b01c6-8001-49e3-908a-38cffb88edfa\" (UID: \"cf2b01c6-8001-49e3-908a-38cffb88edfa\") " Nov 03 00:20:10.952673 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:10.952653 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5cd8831-584f-47bb-9580-4d41deab29dc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5cd8831-584f-47bb-9580-4d41deab29dc-tmp-dir\") pod \"c5cd8831-584f-47bb-9580-4d41deab29dc\" (UID: \"c5cd8831-584f-47bb-9580-4d41deab29dc\") " Nov 03 00:20:10.952949 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:10.952711 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf2b01c6-8001-49e3-908a-38cffb88edfa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf2b01c6-8001-49e3-908a-38cffb88edfa-dshm\") pod \"cf2b01c6-8001-49e3-908a-38cffb88edfa\" (UID: \"cf2b01c6-8001-49e3-908a-38cffb88edfa\") " Nov 03 00:20:10.952949 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:10.952743 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5cd8831-584f-47bb-9580-4d41deab29dc-home\" (UniqueName: \"kubernetes.io/empty-dir/c5cd8831-584f-47bb-9580-4d41deab29dc-home\") pod \"c5cd8831-584f-47bb-9580-4d41deab29dc\" (UID: \"c5cd8831-584f-47bb-9580-4d41deab29dc\") " Nov 03 00:20:10.952949 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:10.952785 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf2b01c6-8001-49e3-908a-38cffb88edfa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf2b01c6-8001-49e3-908a-38cffb88edfa-tmp-dir\") pod \"cf2b01c6-8001-49e3-908a-38cffb88edfa\" (UID: \"cf2b01c6-8001-49e3-908a-38cffb88edfa\") " Nov 03 00:20:10.952949 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:10.952828 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5cd8831-584f-47bb-9580-4d41deab29dc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5cd8831-584f-47bb-9580-4d41deab29dc-model-cache\") pod \"c5cd8831-584f-47bb-9580-4d41deab29dc\" (UID: \"c5cd8831-584f-47bb-9580-4d41deab29dc\") " Nov 03 00:20:10.952949 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:10.952832 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2b01c6-8001-49e3-908a-38cffb88edfa-model-cache" pod "cf2b01c6-8001-49e3-908a-38cffb88edfa" (UID: "cf2b01c6-8001-49e3-908a-38cffb88edfa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:20:10.952949 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:10.952873 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5cd8831-584f-47bb-9580-4d41deab29dc-tls-certs\" (UniqueName: \"kubernetes.io/secret/c5cd8831-584f-47bb-9580-4d41deab29dc-tls-certs\") pod \"c5cd8831-584f-47bb-9580-4d41deab29dc\" (UID: \"c5cd8831-584f-47bb-9580-4d41deab29dc\") " Nov 03 00:20:10.952949 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:10.952939 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf2b01c6-8001-49e3-908a-38cffb88edfa-home\" (UniqueName: \"kubernetes.io/empty-dir/cf2b01c6-8001-49e3-908a-38cffb88edfa-home\") pod \"cf2b01c6-8001-49e3-908a-38cffb88edfa\" (UID: \"cf2b01c6-8001-49e3-908a-38cffb88edfa\") " Nov 03 00:20:10.953349 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:10.952994 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5cd8831-584f-47bb-9580-4d41deab29dc-kube-api-access-xfxw6\" (UniqueName: \"kubernetes.io/projected/c5cd8831-584f-47bb-9580-4d41deab29dc-kube-api-access-xfxw6\") pod \"c5cd8831-584f-47bb-9580-4d41deab29dc\" (UID: \"c5cd8831-584f-47bb-9580-4d41deab29dc\") " Nov 03 00:20:10.953349 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:10.953241 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5cd8831-584f-47bb-9580-4d41deab29dc-home" pod "c5cd8831-584f-47bb-9580-4d41deab29dc" (UID: "c5cd8831-584f-47bb-9580-4d41deab29dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:20:10.953349 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:10.953269 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5cd8831-584f-47bb-9580-4d41deab29dc-tmp-dir" pod "c5cd8831-584f-47bb-9580-4d41deab29dc" (UID: "c5cd8831-584f-47bb-9580-4d41deab29dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:20:10.953520 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:10.953492 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5cd8831-584f-47bb-9580-4d41deab29dc-model-cache" pod "c5cd8831-584f-47bb-9580-4d41deab29dc" (UID: "c5cd8831-584f-47bb-9580-4d41deab29dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:20:10.953520 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:10.953506 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2b01c6-8001-49e3-908a-38cffb88edfa-home" pod "cf2b01c6-8001-49e3-908a-38cffb88edfa" (UID: "cf2b01c6-8001-49e3-908a-38cffb88edfa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:20:10.953639 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:10.953548 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2b01c6-8001-49e3-908a-38cffb88edfa-tmp-dir" pod "cf2b01c6-8001-49e3-908a-38cffb88edfa" (UID: "cf2b01c6-8001-49e3-908a-38cffb88edfa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:20:10.953762 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:10.953742 5076 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5cd8831-584f-47bb-9580-4d41deab29dc-home\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:10.953834 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:10.953767 5076 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf2b01c6-8001-49e3-908a-38cffb88edfa-tmp-dir\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:10.953834 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:10.953793 5076 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5cd8831-584f-47bb-9580-4d41deab29dc-model-cache\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:10.953834 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:10.953808 5076 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf2b01c6-8001-49e3-908a-38cffb88edfa-home\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:10.953834 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:10.953822 5076 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf2b01c6-8001-49e3-908a-38cffb88edfa-model-cache\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:10.954074 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:10.953836 5076 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5cd8831-584f-47bb-9580-4d41deab29dc-tmp-dir\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:10.955452 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:10.955429 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf2b01c6-8001-49e3-908a-38cffb88edfa-tls-certs" pod "cf2b01c6-8001-49e3-908a-38cffb88edfa" (UID: "cf2b01c6-8001-49e3-908a-38cffb88edfa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:20:10.955831 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:10.955806 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5cd8831-584f-47bb-9580-4d41deab29dc-kube-api-access-xfxw6" pod "c5cd8831-584f-47bb-9580-4d41deab29dc" (UID: "c5cd8831-584f-47bb-9580-4d41deab29dc"). InnerVolumeSpecName "kube-api-access-xfxw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:20:10.955967 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:10.955944 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5cd8831-584f-47bb-9580-4d41deab29dc-tls-certs" pod "c5cd8831-584f-47bb-9580-4d41deab29dc" (UID: "c5cd8831-584f-47bb-9580-4d41deab29dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:20:10.956093 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:10.956067 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf2b01c6-8001-49e3-908a-38cffb88edfa-kube-api-access-pwdsg" pod "cf2b01c6-8001-49e3-908a-38cffb88edfa" (UID: "cf2b01c6-8001-49e3-908a-38cffb88edfa"). InnerVolumeSpecName "kube-api-access-pwdsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:20:10.956192 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:10.956113 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5cd8831-584f-47bb-9580-4d41deab29dc-dshm" pod "c5cd8831-584f-47bb-9580-4d41deab29dc" (UID: "c5cd8831-584f-47bb-9580-4d41deab29dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:20:10.956192 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:10.956139 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2b01c6-8001-49e3-908a-38cffb88edfa-dshm" pod "cf2b01c6-8001-49e3-908a-38cffb88edfa" (UID: "cf2b01c6-8001-49e3-908a-38cffb88edfa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:20:11.055125 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:11.055040 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwdsg\" (UniqueName: \"kubernetes.io/projected/cf2b01c6-8001-49e3-908a-38cffb88edfa-kube-api-access-pwdsg\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:11.055125 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:11.055074 5076 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5cd8831-584f-47bb-9580-4d41deab29dc-dshm\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:11.055125 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:11.055085 5076 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf2b01c6-8001-49e3-908a-38cffb88edfa-tls-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:11.055125 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:11.055094 5076 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf2b01c6-8001-49e3-908a-38cffb88edfa-dshm\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:11.055125 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:11.055103 5076 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5cd8831-584f-47bb-9580-4d41deab29dc-tls-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:11.055125 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:11.055112 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfxw6\" (UniqueName: \"kubernetes.io/projected/c5cd8831-584f-47bb-9580-4d41deab29dc-kube-api-access-xfxw6\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:11.520748 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:11.520715 5076 generic.go:356] "Generic (PLEG): container finished" podID="cf2b01c6-8001-49e3-908a-38cffb88edfa" containerID="161090faabb34d26de936b2a96fe2e49a0fbb1cf60a50ca1b8442b78ec1ae00a" exitCode=0 Nov 03 00:20:11.520956 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:11.520772 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" Nov 03 00:20:11.520956 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:11.520799 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" event={"ID":"cf2b01c6-8001-49e3-908a-38cffb88edfa","Type":"ContainerDied","Data":"161090faabb34d26de936b2a96fe2e49a0fbb1cf60a50ca1b8442b78ec1ae00a"} Nov 03 00:20:11.520956 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:11.520839 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" event={"ID":"cf2b01c6-8001-49e3-908a-38cffb88edfa","Type":"ContainerDied","Data":"50f618a8a037feb8fca96822140adea3ff1a08c4549bbe8f7c77f4b907b5da8b"} Nov 03 00:20:11.520956 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:11.520863 5076 scope.go:122] "RemoveContainer" containerID="161090faabb34d26de936b2a96fe2e49a0fbb1cf60a50ca1b8442b78ec1ae00a" Nov 03 00:20:11.522288 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:11.522264 5076 generic.go:356] "Generic (PLEG): container finished" podID="c5cd8831-584f-47bb-9580-4d41deab29dc" containerID="cf6208e292a38fc8dda1168ff8fe0f6913b3ffd612d5ba88a732ae89136a2ad8" exitCode=0 Nov 03 00:20:11.522408 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:11.522300 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" Nov 03 00:20:11.522408 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:11.522344 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" event={"ID":"c5cd8831-584f-47bb-9580-4d41deab29dc","Type":"ContainerDied","Data":"cf6208e292a38fc8dda1168ff8fe0f6913b3ffd612d5ba88a732ae89136a2ad8"} Nov 03 00:20:11.522408 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:11.522381 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv" event={"ID":"c5cd8831-584f-47bb-9580-4d41deab29dc","Type":"ContainerDied","Data":"bcedc58f7537d733ad04b57bcb750bf880f8f2ae3ca9b72235107c203cc4b5df"} Nov 03 00:20:11.529623 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:11.529604 5076 scope.go:122] "RemoveContainer" containerID="161090faabb34d26de936b2a96fe2e49a0fbb1cf60a50ca1b8442b78ec1ae00a" Nov 03 00:20:11.529978 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:20:11.529867 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"161090faabb34d26de936b2a96fe2e49a0fbb1cf60a50ca1b8442b78ec1ae00a\": container with ID starting with 161090faabb34d26de936b2a96fe2e49a0fbb1cf60a50ca1b8442b78ec1ae00a not found: ID does not exist" containerID="161090faabb34d26de936b2a96fe2e49a0fbb1cf60a50ca1b8442b78ec1ae00a" Nov 03 00:20:11.529978 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:11.529913 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"161090faabb34d26de936b2a96fe2e49a0fbb1cf60a50ca1b8442b78ec1ae00a"} err="failed to get container status \"161090faabb34d26de936b2a96fe2e49a0fbb1cf60a50ca1b8442b78ec1ae00a\": rpc error: code = NotFound desc = could not find container \"161090faabb34d26de936b2a96fe2e49a0fbb1cf60a50ca1b8442b78ec1ae00a\": container with ID starting with 161090faabb34d26de936b2a96fe2e49a0fbb1cf60a50ca1b8442b78ec1ae00a not found: ID does not exist" Nov 03 00:20:11.529978 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:11.529938 5076 scope.go:122] "RemoveContainer" containerID="cf6208e292a38fc8dda1168ff8fe0f6913b3ffd612d5ba88a732ae89136a2ad8" Nov 03 00:20:11.537697 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:11.537677 5076 scope.go:122] "RemoveContainer" containerID="cf6208e292a38fc8dda1168ff8fe0f6913b3ffd612d5ba88a732ae89136a2ad8" Nov 03 00:20:11.537985 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:20:11.537965 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf6208e292a38fc8dda1168ff8fe0f6913b3ffd612d5ba88a732ae89136a2ad8\": container with ID starting with cf6208e292a38fc8dda1168ff8fe0f6913b3ffd612d5ba88a732ae89136a2ad8 not found: ID does not exist" containerID="cf6208e292a38fc8dda1168ff8fe0f6913b3ffd612d5ba88a732ae89136a2ad8" Nov 03 00:20:11.538043 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:11.537995 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf6208e292a38fc8dda1168ff8fe0f6913b3ffd612d5ba88a732ae89136a2ad8"} err="failed to get container status \"cf6208e292a38fc8dda1168ff8fe0f6913b3ffd612d5ba88a732ae89136a2ad8\": rpc error: code = NotFound desc = could not find container \"cf6208e292a38fc8dda1168ff8fe0f6913b3ffd612d5ba88a732ae89136a2ad8\": container with ID starting with cf6208e292a38fc8dda1168ff8fe0f6913b3ffd612d5ba88a732ae89136a2ad8 not found: ID does not exist" Nov 03 00:20:11.681784 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:11.681746 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7"] Nov 03 00:20:11.735471 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:11.732254 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7"] Nov 03 00:20:11.794751 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:11.794667 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv"] Nov 03 00:20:11.801181 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:11.801155 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-fb6e1d/premium-w4-fb6e1d-kserve-64d7869d66-8rzmv"] Nov 03 00:20:11.829907 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:11.829855 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-2-w4-fb6e1d-kserve-578d89f8f8-l98d7" podUID="cf2b01c6-8001-49e3-908a-38cffb88edfa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:20:13.146249 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:13.146210 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5cd8831-584f-47bb-9580-4d41deab29dc" path="/var/lib/kubelet/pods/c5cd8831-584f-47bb-9580-4d41deab29dc/volumes" Nov 03 00:20:13.146741 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:13.146650 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf2b01c6-8001-49e3-908a-38cffb88edfa" path="/var/lib/kubelet/pods/cf2b01c6-8001-49e3-908a-38cffb88edfa/volumes" Nov 03 00:20:15.186627 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:15.186592 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4"] Nov 03 00:20:15.187161 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:15.186798 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4" podUID="7485875c-b5a6-4532-833e-80cc5741b3ed" containerName="maas-api" containerID="cri-o://85391d186a6e7276e8491285ade9db1ad12cd78c9184aee5253a94fa0d446d44" gracePeriod=30 Nov 03 00:20:15.446860 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:15.446805 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4" Nov 03 00:20:15.494049 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:15.494019 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7485875c-b5a6-4532-833e-80cc5741b3ed-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7485875c-b5a6-4532-833e-80cc5741b3ed-metrics-certs\") pod \"7485875c-b5a6-4532-833e-80cc5741b3ed\" (UID: \"7485875c-b5a6-4532-833e-80cc5741b3ed\") " Nov 03 00:20:15.494210 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:15.494162 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7485875c-b5a6-4532-833e-80cc5741b3ed-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7485875c-b5a6-4532-833e-80cc5741b3ed-openshift-service-ca\") pod \"7485875c-b5a6-4532-833e-80cc5741b3ed\" (UID: \"7485875c-b5a6-4532-833e-80cc5741b3ed\") " Nov 03 00:20:15.494210 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:15.494185 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7485875c-b5a6-4532-833e-80cc5741b3ed-kube-api-access-mbh2b\" (UniqueName: \"kubernetes.io/projected/7485875c-b5a6-4532-833e-80cc5741b3ed-kube-api-access-mbh2b\") pod \"7485875c-b5a6-4532-833e-80cc5741b3ed\" (UID: \"7485875c-b5a6-4532-833e-80cc5741b3ed\") " Nov 03 00:20:15.494329 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:15.494214 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7485875c-b5a6-4532-833e-80cc5741b3ed-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7485875c-b5a6-4532-833e-80cc5741b3ed-maas-api-tls\") pod \"7485875c-b5a6-4532-833e-80cc5741b3ed\" (UID: \"7485875c-b5a6-4532-833e-80cc5741b3ed\") " Nov 03 00:20:15.494603 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:15.494572 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7485875c-b5a6-4532-833e-80cc5741b3ed-openshift-service-ca" pod "7485875c-b5a6-4532-833e-80cc5741b3ed" (UID: "7485875c-b5a6-4532-833e-80cc5741b3ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:20:15.496175 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:15.496152 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7485875c-b5a6-4532-833e-80cc5741b3ed-metrics-certs" pod "7485875c-b5a6-4532-833e-80cc5741b3ed" (UID: "7485875c-b5a6-4532-833e-80cc5741b3ed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:20:15.496265 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:15.496176 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7485875c-b5a6-4532-833e-80cc5741b3ed-maas-api-tls" pod "7485875c-b5a6-4532-833e-80cc5741b3ed" (UID: "7485875c-b5a6-4532-833e-80cc5741b3ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:20:15.496311 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:15.496267 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7485875c-b5a6-4532-833e-80cc5741b3ed-kube-api-access-mbh2b" pod "7485875c-b5a6-4532-833e-80cc5741b3ed" (UID: "7485875c-b5a6-4532-833e-80cc5741b3ed"). InnerVolumeSpecName "kube-api-access-mbh2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:20:15.550414 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:15.550115 5076 generic.go:356] "Generic (PLEG): container finished" podID="7485875c-b5a6-4532-833e-80cc5741b3ed" containerID="85391d186a6e7276e8491285ade9db1ad12cd78c9184aee5253a94fa0d446d44" exitCode=0 Nov 03 00:20:15.550414 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:15.550175 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4" Nov 03 00:20:15.550414 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:15.550190 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4" event={"ID":"7485875c-b5a6-4532-833e-80cc5741b3ed","Type":"ContainerDied","Data":"85391d186a6e7276e8491285ade9db1ad12cd78c9184aee5253a94fa0d446d44"} Nov 03 00:20:15.550414 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:15.550269 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4" event={"ID":"7485875c-b5a6-4532-833e-80cc5741b3ed","Type":"ContainerDied","Data":"64e87a64bb843525ee908275fff2ee1b6cf7f24e5422ca498da6df8727f0b662"} Nov 03 00:20:15.550414 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:15.550298 5076 scope.go:122] "RemoveContainer" containerID="85391d186a6e7276e8491285ade9db1ad12cd78c9184aee5253a94fa0d446d44" Nov 03 00:20:15.566926 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:15.566901 5076 scope.go:122] "RemoveContainer" containerID="85391d186a6e7276e8491285ade9db1ad12cd78c9184aee5253a94fa0d446d44" Nov 03 00:20:15.567287 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:20:15.567260 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85391d186a6e7276e8491285ade9db1ad12cd78c9184aee5253a94fa0d446d44\": container with ID starting with 85391d186a6e7276e8491285ade9db1ad12cd78c9184aee5253a94fa0d446d44 not found: ID does not exist" containerID="85391d186a6e7276e8491285ade9db1ad12cd78c9184aee5253a94fa0d446d44" Nov 03 00:20:15.567373 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:15.567300 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85391d186a6e7276e8491285ade9db1ad12cd78c9184aee5253a94fa0d446d44"} err="failed to get container status \"85391d186a6e7276e8491285ade9db1ad12cd78c9184aee5253a94fa0d446d44\": rpc error: code = NotFound desc = could not find container \"85391d186a6e7276e8491285ade9db1ad12cd78c9184aee5253a94fa0d446d44\": container with ID starting with 85391d186a6e7276e8491285ade9db1ad12cd78c9184aee5253a94fa0d446d44 not found: ID does not exist" Nov 03 00:20:15.595500 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:15.595473 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7485875c-b5a6-4532-833e-80cc5741b3ed-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:15.595500 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:15.595499 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbh2b\" (UniqueName: \"kubernetes.io/projected/7485875c-b5a6-4532-833e-80cc5741b3ed-kube-api-access-mbh2b\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:15.595669 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:15.595513 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7485875c-b5a6-4532-833e-80cc5741b3ed-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:15.595669 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:15.595522 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7485875c-b5a6-4532-833e-80cc5741b3ed-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:15.717386 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:15.717274 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4"] Nov 03 00:20:15.719400 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:15.719368 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-fb6e1d-78dd4f7594-lcsb4"] Nov 03 00:20:17.146589 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:17.146554 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7485875c-b5a6-4532-833e-80cc5741b3ed" path="/var/lib/kubelet/pods/7485875c-b5a6-4532-833e-80cc5741b3ed/volumes" Nov 03 00:20:24.904986 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.904953 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68"] Nov 03 00:20:24.905473 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905360 5076 cpu_manager.go:414] "containerMap: removing container" podUID="ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2" containerName="istio-proxy" Nov 03 00:20:24.905473 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905371 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2" containerName="istio-proxy" Nov 03 00:20:24.905473 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905387 5076 cpu_manager.go:414] "containerMap: removing container" podUID="9e58f4ff-b1f9-45d4-95ad-e628431da7b7" containerName="cleanup" Nov 03 00:20:24.905473 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905392 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e58f4ff-b1f9-45d4-95ad-e628431da7b7" containerName="cleanup" Nov 03 00:20:24.905473 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905404 5076 cpu_manager.go:414] "containerMap: removing container" podUID="79ab1c7e-f257-492f-ac2f-dfa72b75d711" containerName="maas-api" Nov 03 00:20:24.905473 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905409 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79ab1c7e-f257-492f-ac2f-dfa72b75d711" containerName="maas-api" Nov 03 00:20:24.905473 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905420 5076 cpu_manager.go:414] "containerMap: removing container" podUID="c5cd8831-584f-47bb-9580-4d41deab29dc" containerName="main" Nov 03 00:20:24.905473 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905425 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5cd8831-584f-47bb-9580-4d41deab29dc" containerName="main" Nov 03 00:20:24.905473 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905432 5076 cpu_manager.go:414] "containerMap: removing container" podUID="ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2" containerName="istio-proxy" Nov 03 00:20:24.905473 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905437 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2" containerName="istio-proxy" Nov 03 00:20:24.905473 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905445 5076 cpu_manager.go:414] "containerMap: removing container" podUID="365ba2a8-0d25-4d3e-b6b0-eb577be48fcd" containerName="maas-api" Nov 03 00:20:24.905473 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905450 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="365ba2a8-0d25-4d3e-b6b0-eb577be48fcd" containerName="maas-api" Nov 03 00:20:24.905473 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905458 5076 cpu_manager.go:414] "containerMap: removing container" podUID="cf8cc281-8664-4196-8c74-d0b92d2ae815" containerName="maas-api" Nov 03 00:20:24.905473 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905463 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf8cc281-8664-4196-8c74-d0b92d2ae815" containerName="maas-api" Nov 03 00:20:24.905473 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905469 5076 cpu_manager.go:414] "containerMap: removing container" podUID="6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e" containerName="istio-proxy" Nov 03 00:20:24.905473 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905473 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e" containerName="istio-proxy" Nov 03 00:20:24.905473 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905480 5076 cpu_manager.go:414] "containerMap: removing container" podUID="ffe70026-3a9a-49bb-bc08-e456e35a7c68" containerName="main" Nov 03 00:20:24.906037 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905485 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffe70026-3a9a-49bb-bc08-e456e35a7c68" containerName="main" Nov 03 00:20:24.906037 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905492 5076 cpu_manager.go:414] "containerMap: removing container" podUID="c6827543-3552-4c6c-bfed-203b1081cdc2" containerName="istio-proxy" Nov 03 00:20:24.906037 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905496 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6827543-3552-4c6c-bfed-203b1081cdc2" containerName="istio-proxy" Nov 03 00:20:24.906037 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905504 5076 cpu_manager.go:414] "containerMap: removing container" podUID="f6b0b2ba-6e13-4d79-b2b1-243f80b5703c" containerName="istio-proxy" Nov 03 00:20:24.906037 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905508 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6b0b2ba-6e13-4d79-b2b1-243f80b5703c" containerName="istio-proxy" Nov 03 00:20:24.906037 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905514 5076 cpu_manager.go:414] "containerMap: removing container" podUID="51ba5367-449d-4292-9868-6ff46c0f94d3" containerName="istio-proxy" Nov 03 00:20:24.906037 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905519 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51ba5367-449d-4292-9868-6ff46c0f94d3" containerName="istio-proxy" Nov 03 00:20:24.906037 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905528 5076 cpu_manager.go:414] "containerMap: removing container" podUID="6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e" containerName="istio-proxy" Nov 03 00:20:24.906037 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905532 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e" containerName="istio-proxy" Nov 03 00:20:24.906037 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905542 5076 cpu_manager.go:414] "containerMap: removing container" podUID="f6b0b2ba-6e13-4d79-b2b1-243f80b5703c" containerName="istio-proxy" Nov 03 00:20:24.906037 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905546 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6b0b2ba-6e13-4d79-b2b1-243f80b5703c" containerName="istio-proxy" Nov 03 00:20:24.906037 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905552 5076 cpu_manager.go:414] "containerMap: removing container" podUID="cf2b01c6-8001-49e3-908a-38cffb88edfa" containerName="main" Nov 03 00:20:24.906037 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905556 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf2b01c6-8001-49e3-908a-38cffb88edfa" containerName="main" Nov 03 00:20:24.906037 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905562 5076 cpu_manager.go:414] "containerMap: removing container" podUID="e0ae7fef-665b-4ac5-9028-9214d1a9fe47" containerName="maas-api" Nov 03 00:20:24.906037 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905566 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0ae7fef-665b-4ac5-9028-9214d1a9fe47" containerName="maas-api" Nov 03 00:20:24.906037 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905575 5076 cpu_manager.go:414] "containerMap: removing container" podUID="7485875c-b5a6-4532-833e-80cc5741b3ed" containerName="maas-api" Nov 03 00:20:24.906037 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905579 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7485875c-b5a6-4532-833e-80cc5741b3ed" containerName="maas-api" Nov 03 00:20:24.906037 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905633 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e" containerName="istio-proxy" Nov 03 00:20:24.906037 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905643 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6b0b2ba-6e13-4d79-b2b1-243f80b5703c" containerName="istio-proxy" Nov 03 00:20:24.906037 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905649 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf2b01c6-8001-49e3-908a-38cffb88edfa" containerName="main" Nov 03 00:20:24.906037 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905655 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0ae7fef-665b-4ac5-9028-9214d1a9fe47" containerName="maas-api" Nov 03 00:20:24.906037 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905661 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="7485875c-b5a6-4532-833e-80cc5741b3ed" containerName="maas-api" Nov 03 00:20:24.906037 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905669 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e58f4ff-b1f9-45d4-95ad-e628431da7b7" containerName="cleanup" Nov 03 00:20:24.906037 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905675 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2" containerName="istio-proxy" Nov 03 00:20:24.906037 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905681 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d4868a3-1492-4d61-81ae-080156be2b73" containerName="cleanup" Nov 03 00:20:24.906037 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905687 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="4deff109-12f0-42e8-8908-508e7f6eec68" containerName="istio-proxy" Nov 03 00:20:24.906037 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905694 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="79ab1c7e-f257-492f-ac2f-dfa72b75d711" containerName="maas-api" Nov 03 00:20:24.906037 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905700 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5cd8831-584f-47bb-9580-4d41deab29dc" containerName="main" Nov 03 00:20:24.906037 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905706 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="365ba2a8-0d25-4d3e-b6b0-eb577be48fcd" containerName="maas-api" Nov 03 00:20:24.906037 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905713 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf8cc281-8664-4196-8c74-d0b92d2ae815" containerName="maas-api" Nov 03 00:20:24.906037 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905718 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffe70026-3a9a-49bb-bc08-e456e35a7c68" containerName="main" Nov 03 00:20:24.906037 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905724 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6827543-3552-4c6c-bfed-203b1081cdc2" containerName="istio-proxy" Nov 03 00:20:24.906037 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.905730 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6b0b2ba-6e13-4d79-b2b1-243f80b5703c" containerName="istio-proxy" Nov 03 00:20:24.909274 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.909254 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" Nov 03 00:20:24.981018 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.980984 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bd18470-c641-4889-b29f-834300ef4a84-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68\" (UID: \"6bd18470-c641-4889-b29f-834300ef4a84\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" Nov 03 00:20:24.981187 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.981026 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bd18470-c641-4889-b29f-834300ef4a84-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68\" (UID: \"6bd18470-c641-4889-b29f-834300ef4a84\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" Nov 03 00:20:24.981187 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.981073 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bd18470-c641-4889-b29f-834300ef4a84-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68\" (UID: \"6bd18470-c641-4889-b29f-834300ef4a84\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" Nov 03 00:20:24.981187 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.981162 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bd18470-c641-4889-b29f-834300ef4a84-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68\" (UID: \"6bd18470-c641-4889-b29f-834300ef4a84\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" Nov 03 00:20:24.981300 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.981200 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bd18470-c641-4889-b29f-834300ef4a84-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68\" (UID: \"6bd18470-c641-4889-b29f-834300ef4a84\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" Nov 03 00:20:24.981300 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.981239 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bd18470-c641-4889-b29f-834300ef4a84-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68\" (UID: \"6bd18470-c641-4889-b29f-834300ef4a84\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" Nov 03 00:20:24.981376 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.981348 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd18470-c641-4889-b29f-834300ef4a84-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68\" (UID: \"6bd18470-c641-4889-b29f-834300ef4a84\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" Nov 03 00:20:24.981415 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.981393 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g8qk\" (UniqueName: \"kubernetes.io/projected/6bd18470-c641-4889-b29f-834300ef4a84-kube-api-access-9g8qk\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68\" (UID: \"6bd18470-c641-4889-b29f-834300ef4a84\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" Nov 03 00:20:24.981450 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:24.981423 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd18470-c641-4889-b29f-834300ef4a84-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68\" (UID: \"6bd18470-c641-4889-b29f-834300ef4a84\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" Nov 03 00:20:25.082323 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:25.082288 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd18470-c641-4889-b29f-834300ef4a84-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68\" (UID: \"6bd18470-c641-4889-b29f-834300ef4a84\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" Nov 03 00:20:25.082507 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:25.082339 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g8qk\" (UniqueName: \"kubernetes.io/projected/6bd18470-c641-4889-b29f-834300ef4a84-kube-api-access-9g8qk\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68\" (UID: \"6bd18470-c641-4889-b29f-834300ef4a84\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" Nov 03 00:20:25.082507 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:25.082369 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd18470-c641-4889-b29f-834300ef4a84-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68\" (UID: \"6bd18470-c641-4889-b29f-834300ef4a84\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" Nov 03 00:20:25.082507 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:25.082389 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bd18470-c641-4889-b29f-834300ef4a84-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68\" (UID: \"6bd18470-c641-4889-b29f-834300ef4a84\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" Nov 03 00:20:25.082507 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:25.082413 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bd18470-c641-4889-b29f-834300ef4a84-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68\" (UID: \"6bd18470-c641-4889-b29f-834300ef4a84\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" Nov 03 00:20:25.082507 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:25.082434 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bd18470-c641-4889-b29f-834300ef4a84-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68\" (UID: \"6bd18470-c641-4889-b29f-834300ef4a84\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" Nov 03 00:20:25.082507 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:25.082458 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bd18470-c641-4889-b29f-834300ef4a84-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68\" (UID: \"6bd18470-c641-4889-b29f-834300ef4a84\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" Nov 03 00:20:25.082507 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:25.082492 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bd18470-c641-4889-b29f-834300ef4a84-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68\" (UID: \"6bd18470-c641-4889-b29f-834300ef4a84\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" Nov 03 00:20:25.082929 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:25.082544 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bd18470-c641-4889-b29f-834300ef4a84-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68\" (UID: \"6bd18470-c641-4889-b29f-834300ef4a84\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" Nov 03 00:20:25.082929 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:25.082737 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd18470-c641-4889-b29f-834300ef4a84-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68\" (UID: \"6bd18470-c641-4889-b29f-834300ef4a84\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" Nov 03 00:20:25.082929 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:25.082793 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bd18470-c641-4889-b29f-834300ef4a84-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68\" (UID: \"6bd18470-c641-4889-b29f-834300ef4a84\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" Nov 03 00:20:25.083077 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:25.082932 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bd18470-c641-4889-b29f-834300ef4a84-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68\" (UID: \"6bd18470-c641-4889-b29f-834300ef4a84\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" Nov 03 00:20:25.083077 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:25.082972 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd18470-c641-4889-b29f-834300ef4a84-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68\" (UID: \"6bd18470-c641-4889-b29f-834300ef4a84\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" Nov 03 00:20:25.083312 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:25.083292 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bd18470-c641-4889-b29f-834300ef4a84-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68\" (UID: \"6bd18470-c641-4889-b29f-834300ef4a84\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" Nov 03 00:20:25.084857 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:25.084836 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bd18470-c641-4889-b29f-834300ef4a84-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68\" (UID: \"6bd18470-c641-4889-b29f-834300ef4a84\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" Nov 03 00:20:25.085000 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:25.084981 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bd18470-c641-4889-b29f-834300ef4a84-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68\" (UID: \"6bd18470-c641-4889-b29f-834300ef4a84\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" Nov 03 00:20:25.094273 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:25.094247 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68"] Nov 03 00:20:25.136907 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:25.136865 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bd18470-c641-4889-b29f-834300ef4a84-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68\" (UID: \"6bd18470-c641-4889-b29f-834300ef4a84\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" Nov 03 00:20:25.137074 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:25.137054 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g8qk\" (UniqueName: \"kubernetes.io/projected/6bd18470-c641-4889-b29f-834300ef4a84-kube-api-access-9g8qk\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68\" (UID: \"6bd18470-c641-4889-b29f-834300ef4a84\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" Nov 03 00:20:25.222782 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:25.222751 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" Nov 03 00:20:25.637884 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:25.637853 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68"] Nov 03 00:20:25.639941 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:20:25.639909 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bd18470_c641_4889_b29f_834300ef4a84.slice/crio-3296633e397783c811ca2403482c5ea605e75059ad9d84c2c7dd68e64fa5fc95 WatchSource:0}: Error finding container 3296633e397783c811ca2403482c5ea605e75059ad9d84c2c7dd68e64fa5fc95: Status 404 returned error can't find the container with id 3296633e397783c811ca2403482c5ea605e75059ad9d84c2c7dd68e64fa5fc95 Nov 03 00:20:25.642019 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:25.641982 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:20:25.642136 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:25.642063 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:20:25.642136 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:25.642111 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:20:26.605175 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:26.605136 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" event={"ID":"6bd18470-c641-4889-b29f-834300ef4a84","Type":"ContainerStarted","Data":"85ee2c5e1507aa9acc03fe571dfbb34f32c94e5365490ced790b624fb59edc10"} Nov 03 00:20:26.605564 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:26.605190 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" event={"ID":"6bd18470-c641-4889-b29f-834300ef4a84","Type":"ContainerStarted","Data":"3296633e397783c811ca2403482c5ea605e75059ad9d84c2c7dd68e64fa5fc95"} Nov 03 00:20:26.766684 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:26.766622 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" podStartSLOduration=2.766604176 podStartE2EDuration="2.766604176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:26.765081768 +0000 UTC m=+5275.733613566" watchObservedRunningTime="2026-11-03 00:20:26.766604176 +0000 UTC m=+5275.735135959" Nov 03 00:20:27.230923 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:27.229966 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" Nov 03 00:20:27.231137 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:27.230984 5076 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 03 00:20:27.231137 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:27.231039 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" podUID="6bd18470-c641-4889-b29f-834300ef4a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 03 00:20:28.229374 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:28.229329 5076 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 03 00:20:28.230008 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:28.229974 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" podUID="6bd18470-c641-4889-b29f-834300ef4a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 03 00:20:29.227915 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:29.227350 5076 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 03 00:20:29.227915 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:29.227417 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" podUID="6bd18470-c641-4889-b29f-834300ef4a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 03 00:20:30.229915 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:30.229313 5076 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 03 00:20:30.229915 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:30.229427 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" podUID="6bd18470-c641-4889-b29f-834300ef4a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 03 00:20:31.229934 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:31.229323 5076 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 03 00:20:31.229934 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:31.229393 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" podUID="6bd18470-c641-4889-b29f-834300ef4a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 03 00:20:32.226809 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:32.226778 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" Nov 03 00:20:32.227194 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:32.227165 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" Nov 03 00:20:32.227971 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:32.227948 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" Nov 03 00:20:35.513057 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:35.513021 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68"] Nov 03 00:20:35.513482 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:35.513212 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" podUID="6bd18470-c641-4889-b29f-834300ef4a84" containerName="istio-proxy" containerID="cri-o://85ee2c5e1507aa9acc03fe571dfbb34f32c94e5365490ced790b624fb59edc10" gracePeriod=30 Nov 03 00:20:35.532123 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:35.532092 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg"] Nov 03 00:20:35.532337 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:35.532313 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" podUID="dc8027a0-9041-42ec-af51-46526e00811f" containerName="istio-proxy" containerID="cri-o://51f36e5dc64ab48ba1c7ad1815bd0d6fa057350293cdabaa761e8a07c9cefed7" gracePeriod=30 Nov 03 00:20:37.030702 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:37.030672 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-5rw74_a90148ca-bc21-4a39-8be9-4cff6cc9b34c/manager/0.log" Nov 03 00:20:37.265946 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:37.265911 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-6ff5j_0d983ba6-e119-4cbc-98d2-3dc824673324/manager/0.log" Nov 03 00:20:38.864168 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:38.864137 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-4fvc5_c7e772da-ebf0-4dca-ac74-0c8e4cea112e/manager/0.log" Nov 03 00:20:39.240258 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:39.240223 5076 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:20:39.240436 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:39.240276 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" podUID="dc8027a0-9041-42ec-af51-46526e00811f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:20:39.885019 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:20:39.884980 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:20:40.320703 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.320673 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-hc5jd_5c78102f-a925-4cb0-bc69-971c0d248e4e/manager/0.log" Nov 03 00:20:40.704682 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.704467 5076 generic.go:356] "Generic (PLEG): container finished" podID="dc8027a0-9041-42ec-af51-46526e00811f" containerID="51f36e5dc64ab48ba1c7ad1815bd0d6fa057350293cdabaa761e8a07c9cefed7" exitCode=0 Nov 03 00:20:40.704682 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.704599 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" event={"ID":"dc8027a0-9041-42ec-af51-46526e00811f","Type":"ContainerDied","Data":"51f36e5dc64ab48ba1c7ad1815bd0d6fa057350293cdabaa761e8a07c9cefed7"} Nov 03 00:20:40.707043 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.706858 5076 generic.go:356] "Generic (PLEG): container finished" podID="6bd18470-c641-4889-b29f-834300ef4a84" containerID="85ee2c5e1507aa9acc03fe571dfbb34f32c94e5365490ced790b624fb59edc10" exitCode=0 Nov 03 00:20:40.707043 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.706980 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" event={"ID":"6bd18470-c641-4889-b29f-834300ef4a84","Type":"ContainerDied","Data":"85ee2c5e1507aa9acc03fe571dfbb34f32c94e5365490ced790b624fb59edc10"} Nov 03 00:20:40.786571 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.786546 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" Nov 03 00:20:40.818148 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.818125 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" Nov 03 00:20:40.857482 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.857411 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc8027a0-9041-42ec-af51-46526e00811f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc8027a0-9041-42ec-af51-46526e00811f-workload-certs\") pod \"dc8027a0-9041-42ec-af51-46526e00811f\" (UID: \"dc8027a0-9041-42ec-af51-46526e00811f\") " Nov 03 00:20:40.857482 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.857461 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bd18470-c641-4889-b29f-834300ef4a84-kube-api-access-9g8qk\" (UniqueName: \"kubernetes.io/projected/6bd18470-c641-4889-b29f-834300ef4a84-kube-api-access-9g8qk\") pod \"6bd18470-c641-4889-b29f-834300ef4a84\" (UID: \"6bd18470-c641-4889-b29f-834300ef4a84\") " Nov 03 00:20:40.857681 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.857497 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc8027a0-9041-42ec-af51-46526e00811f-kube-api-access-4krqb\" (UniqueName: \"kubernetes.io/projected/dc8027a0-9041-42ec-af51-46526e00811f-kube-api-access-4krqb\") pod \"dc8027a0-9041-42ec-af51-46526e00811f\" (UID: \"dc8027a0-9041-42ec-af51-46526e00811f\") " Nov 03 00:20:40.857681 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.857538 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bd18470-c641-4889-b29f-834300ef4a84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd18470-c641-4889-b29f-834300ef4a84-workload-socket\") pod \"6bd18470-c641-4889-b29f-834300ef4a84\" (UID: \"6bd18470-c641-4889-b29f-834300ef4a84\") " Nov 03 00:20:40.857681 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.857570 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc8027a0-9041-42ec-af51-46526e00811f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8027a0-9041-42ec-af51-46526e00811f-workload-socket\") pod \"dc8027a0-9041-42ec-af51-46526e00811f\" (UID: \"dc8027a0-9041-42ec-af51-46526e00811f\") " Nov 03 00:20:40.857681 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.857599 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc8027a0-9041-42ec-af51-46526e00811f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc8027a0-9041-42ec-af51-46526e00811f-istio-data\") pod \"dc8027a0-9041-42ec-af51-46526e00811f\" (UID: \"dc8027a0-9041-42ec-af51-46526e00811f\") " Nov 03 00:20:40.857681 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.857631 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc8027a0-9041-42ec-af51-46526e00811f-istio-token\" (UniqueName: \"kubernetes.io/projected/dc8027a0-9041-42ec-af51-46526e00811f-istio-token\") pod \"dc8027a0-9041-42ec-af51-46526e00811f\" (UID: \"dc8027a0-9041-42ec-af51-46526e00811f\") " Nov 03 00:20:40.857681 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.857666 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc8027a0-9041-42ec-af51-46526e00811f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc8027a0-9041-42ec-af51-46526e00811f-istio-envoy\") pod \"dc8027a0-9041-42ec-af51-46526e00811f\" (UID: \"dc8027a0-9041-42ec-af51-46526e00811f\") " Nov 03 00:20:40.858059 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.857697 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc8027a0-9041-42ec-af51-46526e00811f-workload-certs" pod "dc8027a0-9041-42ec-af51-46526e00811f" (UID: "dc8027a0-9041-42ec-af51-46526e00811f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:20:40.858059 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.857714 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc8027a0-9041-42ec-af51-46526e00811f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc8027a0-9041-42ec-af51-46526e00811f-istio-podinfo\") pod \"dc8027a0-9041-42ec-af51-46526e00811f\" (UID: \"dc8027a0-9041-42ec-af51-46526e00811f\") " Nov 03 00:20:40.858059 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.857788 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd18470-c641-4889-b29f-834300ef4a84-workload-socket" pod "6bd18470-c641-4889-b29f-834300ef4a84" (UID: "6bd18470-c641-4889-b29f-834300ef4a84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:20:40.858059 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.857796 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bd18470-c641-4889-b29f-834300ef4a84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd18470-c641-4889-b29f-834300ef4a84-credential-socket\") pod \"6bd18470-c641-4889-b29f-834300ef4a84\" (UID: \"6bd18470-c641-4889-b29f-834300ef4a84\") " Nov 03 00:20:40.858059 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.857842 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6bd18470-c641-4889-b29f-834300ef4a84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bd18470-c641-4889-b29f-834300ef4a84-istio-podinfo\") pod \"6bd18470-c641-4889-b29f-834300ef4a84\" (UID: \"6bd18470-c641-4889-b29f-834300ef4a84\") " Nov 03 00:20:40.858059 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.857870 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bd18470-c641-4889-b29f-834300ef4a84-istio-token\" (UniqueName: \"kubernetes.io/projected/6bd18470-c641-4889-b29f-834300ef4a84-istio-token\") pod \"6bd18470-c641-4889-b29f-834300ef4a84\" (UID: \"6bd18470-c641-4889-b29f-834300ef4a84\") " Nov 03 00:20:40.858059 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.857936 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc8027a0-9041-42ec-af51-46526e00811f-workload-socket" pod "dc8027a0-9041-42ec-af51-46526e00811f" (UID: "dc8027a0-9041-42ec-af51-46526e00811f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:20:40.858059 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.857944 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bd18470-c641-4889-b29f-834300ef4a84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bd18470-c641-4889-b29f-834300ef4a84-workload-certs\") pod \"6bd18470-c641-4889-b29f-834300ef4a84\" (UID: \"6bd18470-c641-4889-b29f-834300ef4a84\") " Nov 03 00:20:40.858059 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.857956 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc8027a0-9041-42ec-af51-46526e00811f-istio-data" pod "dc8027a0-9041-42ec-af51-46526e00811f" (UID: "dc8027a0-9041-42ec-af51-46526e00811f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:20:40.858059 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.857978 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc8027a0-9041-42ec-af51-46526e00811f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8027a0-9041-42ec-af51-46526e00811f-credential-socket\") pod \"dc8027a0-9041-42ec-af51-46526e00811f\" (UID: \"dc8027a0-9041-42ec-af51-46526e00811f\") " Nov 03 00:20:40.858059 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.858021 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc8027a0-9041-42ec-af51-46526e00811f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc8027a0-9041-42ec-af51-46526e00811f-istiod-ca-cert\") pod \"dc8027a0-9041-42ec-af51-46526e00811f\" (UID: \"dc8027a0-9041-42ec-af51-46526e00811f\") " Nov 03 00:20:40.858684 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.858078 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bd18470-c641-4889-b29f-834300ef4a84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bd18470-c641-4889-b29f-834300ef4a84-istiod-ca-cert\") pod \"6bd18470-c641-4889-b29f-834300ef4a84\" (UID: \"6bd18470-c641-4889-b29f-834300ef4a84\") " Nov 03 00:20:40.858684 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.858106 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bd18470-c641-4889-b29f-834300ef4a84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bd18470-c641-4889-b29f-834300ef4a84-istio-envoy\") pod \"6bd18470-c641-4889-b29f-834300ef4a84\" (UID: \"6bd18470-c641-4889-b29f-834300ef4a84\") " Nov 03 00:20:40.858684 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.858147 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bd18470-c641-4889-b29f-834300ef4a84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bd18470-c641-4889-b29f-834300ef4a84-istio-data\") pod \"6bd18470-c641-4889-b29f-834300ef4a84\" (UID: \"6bd18470-c641-4889-b29f-834300ef4a84\") " Nov 03 00:20:40.858684 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.858503 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc8027a0-9041-42ec-af51-46526e00811f-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:40.858684 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.858527 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd18470-c641-4889-b29f-834300ef4a84-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:40.858684 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.858540 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8027a0-9041-42ec-af51-46526e00811f-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:40.858684 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.858553 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc8027a0-9041-42ec-af51-46526e00811f-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:40.858684 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.858497 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd18470-c641-4889-b29f-834300ef4a84-credential-socket" pod "6bd18470-c641-4889-b29f-834300ef4a84" (UID: "6bd18470-c641-4889-b29f-834300ef4a84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:20:40.858684 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.858672 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc8027a0-9041-42ec-af51-46526e00811f-credential-socket" pod "dc8027a0-9041-42ec-af51-46526e00811f" (UID: "dc8027a0-9041-42ec-af51-46526e00811f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:20:40.859219 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.858862 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc8027a0-9041-42ec-af51-46526e00811f-istiod-ca-cert" pod "dc8027a0-9041-42ec-af51-46526e00811f" (UID: "dc8027a0-9041-42ec-af51-46526e00811f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:20:40.859290 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.859227 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bd18470-c641-4889-b29f-834300ef4a84-istiod-ca-cert" pod "6bd18470-c641-4889-b29f-834300ef4a84" (UID: "6bd18470-c641-4889-b29f-834300ef4a84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:20:40.859470 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.859448 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd18470-c641-4889-b29f-834300ef4a84-istio-data" pod "6bd18470-c641-4889-b29f-834300ef4a84" (UID: "6bd18470-c641-4889-b29f-834300ef4a84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:20:40.859785 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.859757 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd18470-c641-4889-b29f-834300ef4a84-workload-certs" pod "6bd18470-c641-4889-b29f-834300ef4a84" (UID: "6bd18470-c641-4889-b29f-834300ef4a84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:20:40.860415 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.860376 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bd18470-c641-4889-b29f-834300ef4a84-kube-api-access-9g8qk" pod "6bd18470-c641-4889-b29f-834300ef4a84" (UID: "6bd18470-c641-4889-b29f-834300ef4a84"). InnerVolumeSpecName "kube-api-access-9g8qk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:20:40.860415 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.860392 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc8027a0-9041-42ec-af51-46526e00811f-istio-token" pod "dc8027a0-9041-42ec-af51-46526e00811f" (UID: "dc8027a0-9041-42ec-af51-46526e00811f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:20:40.860584 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.860566 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc8027a0-9041-42ec-af51-46526e00811f-istio-podinfo" pod "dc8027a0-9041-42ec-af51-46526e00811f" (UID: "dc8027a0-9041-42ec-af51-46526e00811f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:20:40.860679 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.860657 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bd18470-c641-4889-b29f-834300ef4a84-istio-podinfo" pod "6bd18470-c641-4889-b29f-834300ef4a84" (UID: "6bd18470-c641-4889-b29f-834300ef4a84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:20:40.861085 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.861070 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bd18470-c641-4889-b29f-834300ef4a84-istio-token" pod "6bd18470-c641-4889-b29f-834300ef4a84" (UID: "6bd18470-c641-4889-b29f-834300ef4a84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:20:40.861375 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.861346 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc8027a0-9041-42ec-af51-46526e00811f-kube-api-access-4krqb" pod "dc8027a0-9041-42ec-af51-46526e00811f" (UID: "dc8027a0-9041-42ec-af51-46526e00811f"). InnerVolumeSpecName "kube-api-access-4krqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:20:40.861707 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.861693 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd18470-c641-4889-b29f-834300ef4a84-istio-envoy" pod "6bd18470-c641-4889-b29f-834300ef4a84" (UID: "6bd18470-c641-4889-b29f-834300ef4a84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:20:40.861919 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.861874 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc8027a0-9041-42ec-af51-46526e00811f-istio-envoy" pod "dc8027a0-9041-42ec-af51-46526e00811f" (UID: "dc8027a0-9041-42ec-af51-46526e00811f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:20:40.959735 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.959702 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8027a0-9041-42ec-af51-46526e00811f-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:40.959735 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.959733 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc8027a0-9041-42ec-af51-46526e00811f-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:40.959735 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.959743 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bd18470-c641-4889-b29f-834300ef4a84-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:40.960208 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.959751 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bd18470-c641-4889-b29f-834300ef4a84-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:40.960208 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.959760 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bd18470-c641-4889-b29f-834300ef4a84-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:40.960208 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.959768 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g8qk\" (UniqueName: \"kubernetes.io/projected/6bd18470-c641-4889-b29f-834300ef4a84-kube-api-access-9g8qk\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:40.960208 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.959778 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4krqb\" (UniqueName: \"kubernetes.io/projected/dc8027a0-9041-42ec-af51-46526e00811f-kube-api-access-4krqb\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:40.960208 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.959786 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc8027a0-9041-42ec-af51-46526e00811f-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:40.960208 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.959794 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc8027a0-9041-42ec-af51-46526e00811f-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:40.960208 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.959802 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc8027a0-9041-42ec-af51-46526e00811f-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:40.960208 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.959810 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd18470-c641-4889-b29f-834300ef4a84-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:40.960208 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.959818 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bd18470-c641-4889-b29f-834300ef4a84-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:40.960208 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.959827 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bd18470-c641-4889-b29f-834300ef4a84-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:40.960208 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:40.959835 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bd18470-c641-4889-b29f-834300ef4a84-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:41.713185 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:41.713151 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" Nov 03 00:20:41.713185 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:41.713180 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68" event={"ID":"6bd18470-c641-4889-b29f-834300ef4a84","Type":"ContainerDied","Data":"3296633e397783c811ca2403482c5ea605e75059ad9d84c2c7dd68e64fa5fc95"} Nov 03 00:20:41.713421 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:41.713243 5076 scope.go:122] "RemoveContainer" containerID="85ee2c5e1507aa9acc03fe571dfbb34f32c94e5365490ced790b624fb59edc10" Nov 03 00:20:41.714655 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:41.714636 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" event={"ID":"dc8027a0-9041-42ec-af51-46526e00811f","Type":"ContainerDied","Data":"d9ea71ace16e2630edeeeafd9fffdded8e7fdbe3eed15d6c592600c52650d5fb"} Nov 03 00:20:41.714725 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:41.714676 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg" Nov 03 00:20:41.722477 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:41.722453 5076 scope.go:122] "RemoveContainer" containerID="51f36e5dc64ab48ba1c7ad1815bd0d6fa057350293cdabaa761e8a07c9cefed7" Nov 03 00:20:41.917934 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:41.917832 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68"] Nov 03 00:20:41.964729 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:41.964698 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-98z68"] Nov 03 00:20:42.069817 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:42.069784 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg"] Nov 03 00:20:42.071456 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:42.071435 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-dv8cg"] Nov 03 00:20:43.147614 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:43.147582 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bd18470-c641-4889-b29f-834300ef4a84" path="/var/lib/kubelet/pods/6bd18470-c641-4889-b29f-834300ef4a84/volumes" Nov 03 00:20:43.148067 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:43.148051 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc8027a0-9041-42ec-af51-46526e00811f" path="/var/lib/kubelet/pods/dc8027a0-9041-42ec-af51-46526e00811f/volumes" Nov 03 00:20:46.967220 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:46.967183 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s"] Nov 03 00:20:46.967678 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:46.967609 5076 cpu_manager.go:414] "containerMap: removing container" podUID="6bd18470-c641-4889-b29f-834300ef4a84" containerName="istio-proxy" Nov 03 00:20:46.967678 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:46.967625 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bd18470-c641-4889-b29f-834300ef4a84" containerName="istio-proxy" Nov 03 00:20:46.967678 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:46.967649 5076 cpu_manager.go:414] "containerMap: removing container" podUID="dc8027a0-9041-42ec-af51-46526e00811f" containerName="istio-proxy" Nov 03 00:20:46.967678 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:46.967655 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc8027a0-9041-42ec-af51-46526e00811f" containerName="istio-proxy" Nov 03 00:20:46.967807 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:46.967720 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bd18470-c641-4889-b29f-834300ef4a84" containerName="istio-proxy" Nov 03 00:20:46.967807 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:46.967731 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc8027a0-9041-42ec-af51-46526e00811f" containerName="istio-proxy" Nov 03 00:20:46.967807 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:46.967736 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="6caa9d9e-6e17-47ed-bd68-ad6bb8d06f9e" containerName="istio-proxy" Nov 03 00:20:46.967807 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:46.967746 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba4e9d8b-de8c-49e2-a9a1-21f52d7747d2" containerName="istio-proxy" Nov 03 00:20:46.970662 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:46.970645 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" Nov 03 00:20:47.021568 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:47.021534 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a2a3480b-openshift-default-dockercfg-frcw7\"" Nov 03 00:20:47.021852 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:47.021827 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1739f335-a175-4d60-952d-5502f4eb2fb1-istio-token\") pod \"e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s\" (UID: \"1739f335-a175-4d60-952d-5502f4eb2fb1\") " pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" Nov 03 00:20:47.021990 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:47.021867 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1739f335-a175-4d60-952d-5502f4eb2fb1-istiod-ca-cert\") pod \"e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s\" (UID: \"1739f335-a175-4d60-952d-5502f4eb2fb1\") " pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" Nov 03 00:20:47.022037 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:47.021995 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1739f335-a175-4d60-952d-5502f4eb2fb1-istio-data\") pod \"e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s\" (UID: \"1739f335-a175-4d60-952d-5502f4eb2fb1\") " pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" Nov 03 00:20:47.022086 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:47.022038 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1739f335-a175-4d60-952d-5502f4eb2fb1-workload-certs\") pod \"e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s\" (UID: \"1739f335-a175-4d60-952d-5502f4eb2fb1\") " pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" Nov 03 00:20:47.022086 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:47.022068 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5t5z\" (UniqueName: \"kubernetes.io/projected/1739f335-a175-4d60-952d-5502f4eb2fb1-kube-api-access-x5t5z\") pod \"e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s\" (UID: \"1739f335-a175-4d60-952d-5502f4eb2fb1\") " pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" Nov 03 00:20:47.022192 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:47.022174 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1739f335-a175-4d60-952d-5502f4eb2fb1-workload-socket\") pod \"e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s\" (UID: \"1739f335-a175-4d60-952d-5502f4eb2fb1\") " pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" Nov 03 00:20:47.022252 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:47.022219 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1739f335-a175-4d60-952d-5502f4eb2fb1-istio-envoy\") pod \"e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s\" (UID: \"1739f335-a175-4d60-952d-5502f4eb2fb1\") " pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" Nov 03 00:20:47.022305 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:47.022249 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1739f335-a175-4d60-952d-5502f4eb2fb1-credential-socket\") pod \"e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s\" (UID: \"1739f335-a175-4d60-952d-5502f4eb2fb1\") " pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" Nov 03 00:20:47.022371 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:47.022345 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1739f335-a175-4d60-952d-5502f4eb2fb1-istio-podinfo\") pod \"e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s\" (UID: \"1739f335-a175-4d60-952d-5502f4eb2fb1\") " pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" Nov 03 00:20:47.070836 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:47.070802 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s"] Nov 03 00:20:47.123152 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:47.123119 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1739f335-a175-4d60-952d-5502f4eb2fb1-istio-data\") pod \"e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s\" (UID: \"1739f335-a175-4d60-952d-5502f4eb2fb1\") " pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" Nov 03 00:20:47.123335 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:47.123166 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1739f335-a175-4d60-952d-5502f4eb2fb1-workload-certs\") pod \"e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s\" (UID: \"1739f335-a175-4d60-952d-5502f4eb2fb1\") " pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" Nov 03 00:20:47.123335 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:47.123201 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5t5z\" (UniqueName: \"kubernetes.io/projected/1739f335-a175-4d60-952d-5502f4eb2fb1-kube-api-access-x5t5z\") pod \"e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s\" (UID: \"1739f335-a175-4d60-952d-5502f4eb2fb1\") " pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" Nov 03 00:20:47.123510 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:47.123404 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1739f335-a175-4d60-952d-5502f4eb2fb1-workload-socket\") pod \"e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s\" (UID: \"1739f335-a175-4d60-952d-5502f4eb2fb1\") " pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" Nov 03 00:20:47.123606 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:47.123561 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1739f335-a175-4d60-952d-5502f4eb2fb1-istio-envoy\") pod \"e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s\" (UID: \"1739f335-a175-4d60-952d-5502f4eb2fb1\") " pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" Nov 03 00:20:47.123606 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:47.123594 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1739f335-a175-4d60-952d-5502f4eb2fb1-credential-socket\") pod \"e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s\" (UID: \"1739f335-a175-4d60-952d-5502f4eb2fb1\") " pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" Nov 03 00:20:47.123718 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:47.123650 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1739f335-a175-4d60-952d-5502f4eb2fb1-istio-podinfo\") pod \"e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s\" (UID: \"1739f335-a175-4d60-952d-5502f4eb2fb1\") " pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" Nov 03 00:20:47.123718 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:47.123697 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1739f335-a175-4d60-952d-5502f4eb2fb1-istio-token\") pod \"e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s\" (UID: \"1739f335-a175-4d60-952d-5502f4eb2fb1\") " pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" Nov 03 00:20:47.123829 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:47.123724 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1739f335-a175-4d60-952d-5502f4eb2fb1-istiod-ca-cert\") pod \"e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s\" (UID: \"1739f335-a175-4d60-952d-5502f4eb2fb1\") " pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" Nov 03 00:20:47.124586 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:47.124532 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1739f335-a175-4d60-952d-5502f4eb2fb1-workload-socket\") pod \"e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s\" (UID: \"1739f335-a175-4d60-952d-5502f4eb2fb1\") " pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" Nov 03 00:20:47.124877 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:47.124853 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1739f335-a175-4d60-952d-5502f4eb2fb1-istio-data\") pod \"e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s\" (UID: \"1739f335-a175-4d60-952d-5502f4eb2fb1\") " pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" Nov 03 00:20:47.125079 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:47.125057 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1739f335-a175-4d60-952d-5502f4eb2fb1-credential-socket\") pod \"e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s\" (UID: \"1739f335-a175-4d60-952d-5502f4eb2fb1\") " pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" Nov 03 00:20:47.126194 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:47.125201 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1739f335-a175-4d60-952d-5502f4eb2fb1-workload-certs\") pod \"e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s\" (UID: \"1739f335-a175-4d60-952d-5502f4eb2fb1\") " pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" Nov 03 00:20:47.126605 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:47.125835 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1739f335-a175-4d60-952d-5502f4eb2fb1-istiod-ca-cert\") pod \"e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s\" (UID: \"1739f335-a175-4d60-952d-5502f4eb2fb1\") " pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" Nov 03 00:20:47.127240 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:47.127177 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1739f335-a175-4d60-952d-5502f4eb2fb1-istio-envoy\") pod \"e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s\" (UID: \"1739f335-a175-4d60-952d-5502f4eb2fb1\") " pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" Nov 03 00:20:47.127636 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:47.127585 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1739f335-a175-4d60-952d-5502f4eb2fb1-istio-podinfo\") pod \"e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s\" (UID: \"1739f335-a175-4d60-952d-5502f4eb2fb1\") " pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" Nov 03 00:20:47.201701 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:47.201669 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1739f335-a175-4d60-952d-5502f4eb2fb1-istio-token\") pod \"e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s\" (UID: \"1739f335-a175-4d60-952d-5502f4eb2fb1\") " pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" Nov 03 00:20:47.201852 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:47.201795 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5t5z\" (UniqueName: \"kubernetes.io/projected/1739f335-a175-4d60-952d-5502f4eb2fb1-kube-api-access-x5t5z\") pod \"e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s\" (UID: \"1739f335-a175-4d60-952d-5502f4eb2fb1\") " pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" Nov 03 00:20:47.283045 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:47.282974 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" Nov 03 00:20:47.699425 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:47.699396 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s"] Nov 03 00:20:47.701730 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:20:47.701702 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1739f335_a175_4d60_952d_5502f4eb2fb1.slice/crio-35b99a9e3677028855f95b54bafb0d293e328164511b16d83a5103ef8db64653 WatchSource:0}: Error finding container 35b99a9e3677028855f95b54bafb0d293e328164511b16d83a5103ef8db64653: Status 404 returned error can't find the container with id 35b99a9e3677028855f95b54bafb0d293e328164511b16d83a5103ef8db64653 Nov 03 00:20:47.703885 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:47.703853 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:20:47.703995 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:47.703933 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:20:47.703995 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:47.703960 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:20:47.748374 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:47.748347 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" event={"ID":"1739f335-a175-4d60-952d-5502f4eb2fb1","Type":"ContainerStarted","Data":"35b99a9e3677028855f95b54bafb0d293e328164511b16d83a5103ef8db64653"} Nov 03 00:20:48.755229 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:48.755193 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" event={"ID":"1739f335-a175-4d60-952d-5502f4eb2fb1","Type":"ContainerStarted","Data":"d4c0ca1e19a494b7f79ee033f5b540a8652d786008ddc1895dc438450206e8fb"} Nov 03 00:20:49.283812 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:49.283774 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" Nov 03 00:20:49.288861 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:49.288836 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" Nov 03 00:20:49.404275 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:49.404219 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" podStartSLOduration=3.404202333 podStartE2EDuration="3.404202333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:48.920691095 +0000 UTC m=+5297.889222882" watchObservedRunningTime="2026-11-03 00:20:49.404202333 +0000 UTC m=+5298.372734116" Nov 03 00:20:49.760051 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:49.760017 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" Nov 03 00:20:49.761078 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:49.761058 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" Nov 03 00:20:56.775186 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:56.775142 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv"] Nov 03 00:20:56.780929 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:56.780903 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv" Nov 03 00:20:56.878336 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:56.878296 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a2a3480b-serving-cert\"" Nov 03 00:20:56.878336 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:56.878334 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 00:20:56.878572 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:56.878427 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-b7gfc\"" Nov 03 00:20:56.881188 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:56.881165 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv"] Nov 03 00:20:56.914282 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:56.914241 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4ca75f2-4e9b-4771-85cc-f57d3b4404f6-metrics-certs\") pod \"maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv\" (UID: \"d4ca75f2-4e9b-4771-85cc-f57d3b4404f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv" Nov 03 00:20:56.914451 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:56.914385 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jf24\" (UniqueName: \"kubernetes.io/projected/d4ca75f2-4e9b-4771-85cc-f57d3b4404f6-kube-api-access-8jf24\") pod \"maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv\" (UID: \"d4ca75f2-4e9b-4771-85cc-f57d3b4404f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv" Nov 03 00:20:56.914451 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:56.914445 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4ca75f2-4e9b-4771-85cc-f57d3b4404f6-openshift-service-ca\") pod \"maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv\" (UID: \"d4ca75f2-4e9b-4771-85cc-f57d3b4404f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv" Nov 03 00:20:56.914584 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:56.914559 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4ca75f2-4e9b-4771-85cc-f57d3b4404f6-maas-api-tls\") pod \"maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv\" (UID: \"d4ca75f2-4e9b-4771-85cc-f57d3b4404f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv" Nov 03 00:20:57.015097 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:57.015060 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jf24\" (UniqueName: \"kubernetes.io/projected/d4ca75f2-4e9b-4771-85cc-f57d3b4404f6-kube-api-access-8jf24\") pod \"maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv\" (UID: \"d4ca75f2-4e9b-4771-85cc-f57d3b4404f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv" Nov 03 00:20:57.015287 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:57.015119 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4ca75f2-4e9b-4771-85cc-f57d3b4404f6-openshift-service-ca\") pod \"maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv\" (UID: \"d4ca75f2-4e9b-4771-85cc-f57d3b4404f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv" Nov 03 00:20:57.015287 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:57.015153 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4ca75f2-4e9b-4771-85cc-f57d3b4404f6-maas-api-tls\") pod \"maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv\" (UID: \"d4ca75f2-4e9b-4771-85cc-f57d3b4404f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv" Nov 03 00:20:57.015287 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:57.015186 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4ca75f2-4e9b-4771-85cc-f57d3b4404f6-metrics-certs\") pod \"maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv\" (UID: \"d4ca75f2-4e9b-4771-85cc-f57d3b4404f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv" Nov 03 00:20:57.015782 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:57.015750 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4ca75f2-4e9b-4771-85cc-f57d3b4404f6-openshift-service-ca\") pod \"maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv\" (UID: \"d4ca75f2-4e9b-4771-85cc-f57d3b4404f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv" Nov 03 00:20:57.017655 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:57.017634 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4ca75f2-4e9b-4771-85cc-f57d3b4404f6-metrics-certs\") pod \"maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv\" (UID: \"d4ca75f2-4e9b-4771-85cc-f57d3b4404f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv" Nov 03 00:20:57.017765 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:57.017671 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4ca75f2-4e9b-4771-85cc-f57d3b4404f6-maas-api-tls\") pod \"maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv\" (UID: \"d4ca75f2-4e9b-4771-85cc-f57d3b4404f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv" Nov 03 00:20:57.069583 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:57.069503 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jf24\" (UniqueName: \"kubernetes.io/projected/d4ca75f2-4e9b-4771-85cc-f57d3b4404f6-kube-api-access-8jf24\") pod \"maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv\" (UID: \"d4ca75f2-4e9b-4771-85cc-f57d3b4404f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv" Nov 03 00:20:57.096127 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:57.096099 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv" Nov 03 00:20:57.461236 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:57.461213 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv"] Nov 03 00:20:57.463267 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:20:57.463234 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4ca75f2_4e9b_4771_85cc_f57d3b4404f6.slice/crio-15efe35b7a41f4b5e3e0dc70f52afcc45681a9e0be502720300d82638733269f WatchSource:0}: Error finding container 15efe35b7a41f4b5e3e0dc70f52afcc45681a9e0be502720300d82638733269f: Status 404 returned error can't find the container with id 15efe35b7a41f4b5e3e0dc70f52afcc45681a9e0be502720300d82638733269f Nov 03 00:20:57.464508 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:57.464487 5076 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:20:57.800332 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:57.800255 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv" event={"ID":"d4ca75f2-4e9b-4771-85cc-f57d3b4404f6","Type":"ContainerStarted","Data":"15efe35b7a41f4b5e3e0dc70f52afcc45681a9e0be502720300d82638733269f"} Nov 03 00:20:58.807562 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:58.807461 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv" event={"ID":"d4ca75f2-4e9b-4771-85cc-f57d3b4404f6","Type":"ContainerStarted","Data":"4206b83ee3365be06a19e706f94775ab304e57545317be266c91068b5739473b"} Nov 03 00:20:58.807975 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:58.807570 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv" Nov 03 00:20:58.919246 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:20:58.919187 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv" podStartSLOduration=1.895718525 podStartE2EDuration="2.919173286s" totalImagesPullingTime="1.023454761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:58.91779525 +0000 UTC m=+5307.886327046" watchObservedRunningTime="2026-11-03 00:20:58.919173286 +0000 UTC m=+5307.887705070" Nov 03 00:21:04.818337 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:04.818304 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv" Nov 03 00:21:09.886154 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:21:09.886113 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:21:14.890037 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:14.890002 5076 generic.go:356] "Generic (PLEG): container finished" podID="1739f335-a175-4d60-952d-5502f4eb2fb1" containerID="d4c0ca1e19a494b7f79ee033f5b540a8652d786008ddc1895dc438450206e8fb" exitCode=0 Nov 03 00:21:14.890498 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:14.890073 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" event={"ID":"1739f335-a175-4d60-952d-5502f4eb2fb1","Type":"ContainerDied","Data":"d4c0ca1e19a494b7f79ee033f5b540a8652d786008ddc1895dc438450206e8fb"} Nov 03 00:21:14.890498 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:14.890464 5076 scope.go:122] "RemoveContainer" containerID="d4c0ca1e19a494b7f79ee033f5b540a8652d786008ddc1895dc438450206e8fb" Nov 03 00:21:14.891644 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:14.891619 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:21:14.891730 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:14.891684 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:21:14.891730 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:14.891718 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:21:15.896717 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:15.896683 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" event={"ID":"1739f335-a175-4d60-952d-5502f4eb2fb1","Type":"ContainerStarted","Data":"08cb1b05672b13d0177379c810b515990102386a263476cd02407b04be9e8143"} Nov 03 00:21:16.284224 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:16.284185 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" Nov 03 00:21:16.285495 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:16.285466 5076 patch_prober.go:29] interesting pod/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 00:21:16.285640 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:16.285529 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" podUID="1739f335-a175-4d60-952d-5502f4eb2fb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 00:21:17.283930 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:17.283876 5076 patch_prober.go:29] interesting pod/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 00:21:17.284332 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:17.283962 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" podUID="1739f335-a175-4d60-952d-5502f4eb2fb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 00:21:18.283375 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:18.283338 5076 patch_prober.go:29] interesting pod/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 00:21:18.283573 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:18.283388 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" podUID="1739f335-a175-4d60-952d-5502f4eb2fb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 00:21:19.283554 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:19.283515 5076 patch_prober.go:29] interesting pod/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 00:21:19.284009 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:19.283594 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" podUID="1739f335-a175-4d60-952d-5502f4eb2fb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 00:21:19.760968 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:19.760931 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" Nov 03 00:21:20.283932 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:20.283867 5076 patch_prober.go:29] interesting pod/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 00:21:20.284339 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:20.283953 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" podUID="1739f335-a175-4d60-952d-5502f4eb2fb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 00:21:21.287563 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:21.287526 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" Nov 03 00:21:21.288570 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:21.288552 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" Nov 03 00:21:34.714033 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:34.713989 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv"] Nov 03 00:21:34.714564 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:34.714275 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv" podUID="d4ca75f2-4e9b-4771-85cc-f57d3b4404f6" containerName="maas-api" containerID="cri-o://4206b83ee3365be06a19e706f94775ab304e57545317be266c91068b5739473b" gracePeriod=30 Nov 03 00:21:34.813723 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:34.813687 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv" podUID="d4ca75f2-4e9b-4771-85cc-f57d3b4404f6" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.208:8443/health\": dial tcp 10.132.0.208:8443: connect: connection refused" Nov 03 00:21:34.973042 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:34.972978 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv" Nov 03 00:21:35.017797 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:35.017764 5076 generic.go:356] "Generic (PLEG): container finished" podID="d4ca75f2-4e9b-4771-85cc-f57d3b4404f6" containerID="4206b83ee3365be06a19e706f94775ab304e57545317be266c91068b5739473b" exitCode=0 Nov 03 00:21:35.017988 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:35.017806 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv" Nov 03 00:21:35.017988 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:35.017846 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv" event={"ID":"d4ca75f2-4e9b-4771-85cc-f57d3b4404f6","Type":"ContainerDied","Data":"4206b83ee3365be06a19e706f94775ab304e57545317be266c91068b5739473b"} Nov 03 00:21:35.017988 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:35.017906 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv" event={"ID":"d4ca75f2-4e9b-4771-85cc-f57d3b4404f6","Type":"ContainerDied","Data":"15efe35b7a41f4b5e3e0dc70f52afcc45681a9e0be502720300d82638733269f"} Nov 03 00:21:35.017988 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:35.017931 5076 scope.go:122] "RemoveContainer" containerID="4206b83ee3365be06a19e706f94775ab304e57545317be266c91068b5739473b" Nov 03 00:21:35.026701 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:35.026680 5076 scope.go:122] "RemoveContainer" containerID="4206b83ee3365be06a19e706f94775ab304e57545317be266c91068b5739473b" Nov 03 00:21:35.026922 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:35.026883 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4ca75f2-4e9b-4771-85cc-f57d3b4404f6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4ca75f2-4e9b-4771-85cc-f57d3b4404f6-maas-api-tls\") pod \"d4ca75f2-4e9b-4771-85cc-f57d3b4404f6\" (UID: \"d4ca75f2-4e9b-4771-85cc-f57d3b4404f6\") " Nov 03 00:21:35.027023 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:35.026941 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4ca75f2-4e9b-4771-85cc-f57d3b4404f6-kube-api-access-8jf24\" (UniqueName: \"kubernetes.io/projected/d4ca75f2-4e9b-4771-85cc-f57d3b4404f6-kube-api-access-8jf24\") pod \"d4ca75f2-4e9b-4771-85cc-f57d3b4404f6\" (UID: \"d4ca75f2-4e9b-4771-85cc-f57d3b4404f6\") " Nov 03 00:21:35.027023 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:35.026971 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4ca75f2-4e9b-4771-85cc-f57d3b4404f6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4ca75f2-4e9b-4771-85cc-f57d3b4404f6-metrics-certs\") pod \"d4ca75f2-4e9b-4771-85cc-f57d3b4404f6\" (UID: \"d4ca75f2-4e9b-4771-85cc-f57d3b4404f6\") " Nov 03 00:21:35.027023 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:21:35.026987 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4206b83ee3365be06a19e706f94775ab304e57545317be266c91068b5739473b\": container with ID starting with 4206b83ee3365be06a19e706f94775ab304e57545317be266c91068b5739473b not found: ID does not exist" containerID="4206b83ee3365be06a19e706f94775ab304e57545317be266c91068b5739473b" Nov 03 00:21:35.027199 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:35.027028 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4206b83ee3365be06a19e706f94775ab304e57545317be266c91068b5739473b"} err="failed to get container status \"4206b83ee3365be06a19e706f94775ab304e57545317be266c91068b5739473b\": rpc error: code = NotFound desc = could not find container \"4206b83ee3365be06a19e706f94775ab304e57545317be266c91068b5739473b\": container with ID starting with 4206b83ee3365be06a19e706f94775ab304e57545317be266c91068b5739473b not found: ID does not exist" Nov 03 00:21:35.027199 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:35.027002 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4ca75f2-4e9b-4771-85cc-f57d3b4404f6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4ca75f2-4e9b-4771-85cc-f57d3b4404f6-openshift-service-ca\") pod \"d4ca75f2-4e9b-4771-85cc-f57d3b4404f6\" (UID: \"d4ca75f2-4e9b-4771-85cc-f57d3b4404f6\") " Nov 03 00:21:35.027308 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:35.027291 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4ca75f2-4e9b-4771-85cc-f57d3b4404f6-openshift-service-ca" pod "d4ca75f2-4e9b-4771-85cc-f57d3b4404f6" (UID: "d4ca75f2-4e9b-4771-85cc-f57d3b4404f6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:21:35.027454 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:35.027440 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4ca75f2-4e9b-4771-85cc-f57d3b4404f6-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:21:35.028778 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:35.028758 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4ca75f2-4e9b-4771-85cc-f57d3b4404f6-maas-api-tls" pod "d4ca75f2-4e9b-4771-85cc-f57d3b4404f6" (UID: "d4ca75f2-4e9b-4771-85cc-f57d3b4404f6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:21:35.028970 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:35.028952 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4ca75f2-4e9b-4771-85cc-f57d3b4404f6-kube-api-access-8jf24" pod "d4ca75f2-4e9b-4771-85cc-f57d3b4404f6" (UID: "d4ca75f2-4e9b-4771-85cc-f57d3b4404f6"). InnerVolumeSpecName "kube-api-access-8jf24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:21:35.029064 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:35.029050 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4ca75f2-4e9b-4771-85cc-f57d3b4404f6-metrics-certs" pod "d4ca75f2-4e9b-4771-85cc-f57d3b4404f6" (UID: "d4ca75f2-4e9b-4771-85cc-f57d3b4404f6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:21:35.128691 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:35.128655 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4ca75f2-4e9b-4771-85cc-f57d3b4404f6-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:21:35.128691 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:35.128684 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jf24\" (UniqueName: \"kubernetes.io/projected/d4ca75f2-4e9b-4771-85cc-f57d3b4404f6-kube-api-access-8jf24\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:21:35.128691 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:35.128695 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4ca75f2-4e9b-4771-85cc-f57d3b4404f6-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:21:35.427674 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:35.427586 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv"] Nov 03 00:21:35.434662 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:35.434629 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a2a3480b-6fb9694679-7v4jv"] Nov 03 00:21:37.146624 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:37.146588 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4ca75f2-4e9b-4771-85cc-f57d3b4404f6" path="/var/lib/kubelet/pods/d4ca75f2-4e9b-4771-85cc-f57d3b4404f6/volumes" Nov 03 00:21:38.175725 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:38.175691 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s"] Nov 03 00:21:38.176138 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:38.175917 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" podUID="1739f335-a175-4d60-952d-5502f4eb2fb1" containerName="istio-proxy" containerID="cri-o://08cb1b05672b13d0177379c810b515990102386a263476cd02407b04be9e8143" gracePeriod=30 Nov 03 00:21:39.884377 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:21:39.884339 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:21:43.431305 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.431278 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" Nov 03 00:21:43.521985 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.521879 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1739f335-a175-4d60-952d-5502f4eb2fb1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1739f335-a175-4d60-952d-5502f4eb2fb1-workload-socket\") pod \"1739f335-a175-4d60-952d-5502f4eb2fb1\" (UID: \"1739f335-a175-4d60-952d-5502f4eb2fb1\") " Nov 03 00:21:43.521985 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.521973 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1739f335-a175-4d60-952d-5502f4eb2fb1-istio-token\" (UniqueName: \"kubernetes.io/projected/1739f335-a175-4d60-952d-5502f4eb2fb1-istio-token\") pod \"1739f335-a175-4d60-952d-5502f4eb2fb1\" (UID: \"1739f335-a175-4d60-952d-5502f4eb2fb1\") " Nov 03 00:21:43.522218 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.521997 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1739f335-a175-4d60-952d-5502f4eb2fb1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1739f335-a175-4d60-952d-5502f4eb2fb1-workload-certs\") pod \"1739f335-a175-4d60-952d-5502f4eb2fb1\" (UID: \"1739f335-a175-4d60-952d-5502f4eb2fb1\") " Nov 03 00:21:43.522218 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.522041 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1739f335-a175-4d60-952d-5502f4eb2fb1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1739f335-a175-4d60-952d-5502f4eb2fb1-istio-data\") pod \"1739f335-a175-4d60-952d-5502f4eb2fb1\" (UID: \"1739f335-a175-4d60-952d-5502f4eb2fb1\") " Nov 03 00:21:43.522218 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.522096 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1739f335-a175-4d60-952d-5502f4eb2fb1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1739f335-a175-4d60-952d-5502f4eb2fb1-istio-podinfo\") pod \"1739f335-a175-4d60-952d-5502f4eb2fb1\" (UID: \"1739f335-a175-4d60-952d-5502f4eb2fb1\") " Nov 03 00:21:43.522218 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.522138 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1739f335-a175-4d60-952d-5502f4eb2fb1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1739f335-a175-4d60-952d-5502f4eb2fb1-credential-socket\") pod \"1739f335-a175-4d60-952d-5502f4eb2fb1\" (UID: \"1739f335-a175-4d60-952d-5502f4eb2fb1\") " Nov 03 00:21:43.522218 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.522155 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1739f335-a175-4d60-952d-5502f4eb2fb1-workload-socket" pod "1739f335-a175-4d60-952d-5502f4eb2fb1" (UID: "1739f335-a175-4d60-952d-5502f4eb2fb1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:21:43.522218 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.522183 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1739f335-a175-4d60-952d-5502f4eb2fb1-kube-api-access-x5t5z\" (UniqueName: \"kubernetes.io/projected/1739f335-a175-4d60-952d-5502f4eb2fb1-kube-api-access-x5t5z\") pod \"1739f335-a175-4d60-952d-5502f4eb2fb1\" (UID: \"1739f335-a175-4d60-952d-5502f4eb2fb1\") " Nov 03 00:21:43.522218 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.522210 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1739f335-a175-4d60-952d-5502f4eb2fb1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1739f335-a175-4d60-952d-5502f4eb2fb1-istiod-ca-cert\") pod \"1739f335-a175-4d60-952d-5502f4eb2fb1\" (UID: \"1739f335-a175-4d60-952d-5502f4eb2fb1\") " Nov 03 00:21:43.523030 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.522263 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1739f335-a175-4d60-952d-5502f4eb2fb1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1739f335-a175-4d60-952d-5502f4eb2fb1-istio-envoy\") pod \"1739f335-a175-4d60-952d-5502f4eb2fb1\" (UID: \"1739f335-a175-4d60-952d-5502f4eb2fb1\") " Nov 03 00:21:43.523030 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.522323 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1739f335-a175-4d60-952d-5502f4eb2fb1-workload-certs" pod "1739f335-a175-4d60-952d-5502f4eb2fb1" (UID: "1739f335-a175-4d60-952d-5502f4eb2fb1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:21:43.523030 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.522529 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1739f335-a175-4d60-952d-5502f4eb2fb1-credential-socket" pod "1739f335-a175-4d60-952d-5502f4eb2fb1" (UID: "1739f335-a175-4d60-952d-5502f4eb2fb1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:21:43.523030 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.522671 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1739f335-a175-4d60-952d-5502f4eb2fb1-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:21:43.523030 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.522687 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1739f335-a175-4d60-952d-5502f4eb2fb1-istio-data" pod "1739f335-a175-4d60-952d-5502f4eb2fb1" (UID: "1739f335-a175-4d60-952d-5502f4eb2fb1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:21:43.523030 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.522693 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1739f335-a175-4d60-952d-5502f4eb2fb1-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:21:43.523030 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.522725 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1739f335-a175-4d60-952d-5502f4eb2fb1-istiod-ca-cert" pod "1739f335-a175-4d60-952d-5502f4eb2fb1" (UID: "1739f335-a175-4d60-952d-5502f4eb2fb1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:21:43.523030 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.522736 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1739f335-a175-4d60-952d-5502f4eb2fb1-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:21:43.524755 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.524730 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1739f335-a175-4d60-952d-5502f4eb2fb1-istio-podinfo" pod "1739f335-a175-4d60-952d-5502f4eb2fb1" (UID: "1739f335-a175-4d60-952d-5502f4eb2fb1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:21:43.524867 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.524736 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1739f335-a175-4d60-952d-5502f4eb2fb1-istio-token" pod "1739f335-a175-4d60-952d-5502f4eb2fb1" (UID: "1739f335-a175-4d60-952d-5502f4eb2fb1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:21:43.524867 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.524846 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1739f335-a175-4d60-952d-5502f4eb2fb1-istio-envoy" pod "1739f335-a175-4d60-952d-5502f4eb2fb1" (UID: "1739f335-a175-4d60-952d-5502f4eb2fb1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:21:43.524867 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.524856 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1739f335-a175-4d60-952d-5502f4eb2fb1-kube-api-access-x5t5z" pod "1739f335-a175-4d60-952d-5502f4eb2fb1" (UID: "1739f335-a175-4d60-952d-5502f4eb2fb1"). InnerVolumeSpecName "kube-api-access-x5t5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:21:43.623777 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.623740 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5t5z\" (UniqueName: \"kubernetes.io/projected/1739f335-a175-4d60-952d-5502f4eb2fb1-kube-api-access-x5t5z\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:21:43.623777 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.623771 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1739f335-a175-4d60-952d-5502f4eb2fb1-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:21:43.623777 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.623781 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1739f335-a175-4d60-952d-5502f4eb2fb1-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:21:43.624053 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.623789 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1739f335-a175-4d60-952d-5502f4eb2fb1-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:21:43.624053 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.623799 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1739f335-a175-4d60-952d-5502f4eb2fb1-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:21:43.624053 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.623807 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1739f335-a175-4d60-952d-5502f4eb2fb1-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:21:43.796939 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.796839 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-b67978596-6r6f9"] Nov 03 00:21:43.797265 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.797251 5076 cpu_manager.go:414] "containerMap: removing container" podUID="1739f335-a175-4d60-952d-5502f4eb2fb1" containerName="istio-proxy" Nov 03 00:21:43.797318 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.797268 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1739f335-a175-4d60-952d-5502f4eb2fb1" containerName="istio-proxy" Nov 03 00:21:43.797318 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.797293 5076 cpu_manager.go:414] "containerMap: removing container" podUID="d4ca75f2-4e9b-4771-85cc-f57d3b4404f6" containerName="maas-api" Nov 03 00:21:43.797318 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.797299 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4ca75f2-4e9b-4771-85cc-f57d3b4404f6" containerName="maas-api" Nov 03 00:21:43.797421 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.797371 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4ca75f2-4e9b-4771-85cc-f57d3b4404f6" containerName="maas-api" Nov 03 00:21:43.797421 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.797385 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="1739f335-a175-4d60-952d-5502f4eb2fb1" containerName="istio-proxy" Nov 03 00:21:43.797421 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.797392 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="1739f335-a175-4d60-952d-5502f4eb2fb1" containerName="istio-proxy" Nov 03 00:21:43.800696 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.800677 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b67978596-6r6f9" Nov 03 00:21:43.854573 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.854523 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 00:21:43.854733 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.854623 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 00:21:43.854809 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.854759 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 00:21:43.854958 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.854940 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8fxqx\"" Nov 03 00:21:43.857253 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.857228 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b67978596-6r6f9"] Nov 03 00:21:43.926302 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.926269 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld8x4\" (UniqueName: \"kubernetes.io/projected/2ceea576-7104-4b64-a0c2-a68ba70cabb8-kube-api-access-ld8x4\") pod \"maas-controller-b67978596-6r6f9\" (UID: \"2ceea576-7104-4b64-a0c2-a68ba70cabb8\") " pod="opendatahub/maas-controller-b67978596-6r6f9" Nov 03 00:21:43.926445 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.926317 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ceea576-7104-4b64-a0c2-a68ba70cabb8-webhook-cert\") pod \"maas-controller-b67978596-6r6f9\" (UID: \"2ceea576-7104-4b64-a0c2-a68ba70cabb8\") " pod="opendatahub/maas-controller-b67978596-6r6f9" Nov 03 00:21:43.926445 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:43.926345 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ceea576-7104-4b64-a0c2-a68ba70cabb8-metrics-certs\") pod \"maas-controller-b67978596-6r6f9\" (UID: \"2ceea576-7104-4b64-a0c2-a68ba70cabb8\") " pod="opendatahub/maas-controller-b67978596-6r6f9" Nov 03 00:21:44.026937 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:44.026877 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld8x4\" (UniqueName: \"kubernetes.io/projected/2ceea576-7104-4b64-a0c2-a68ba70cabb8-kube-api-access-ld8x4\") pod \"maas-controller-b67978596-6r6f9\" (UID: \"2ceea576-7104-4b64-a0c2-a68ba70cabb8\") " pod="opendatahub/maas-controller-b67978596-6r6f9" Nov 03 00:21:44.027131 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:44.026962 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ceea576-7104-4b64-a0c2-a68ba70cabb8-webhook-cert\") pod \"maas-controller-b67978596-6r6f9\" (UID: \"2ceea576-7104-4b64-a0c2-a68ba70cabb8\") " pod="opendatahub/maas-controller-b67978596-6r6f9" Nov 03 00:21:44.027131 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:44.026988 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ceea576-7104-4b64-a0c2-a68ba70cabb8-metrics-certs\") pod \"maas-controller-b67978596-6r6f9\" (UID: \"2ceea576-7104-4b64-a0c2-a68ba70cabb8\") " pod="opendatahub/maas-controller-b67978596-6r6f9" Nov 03 00:21:44.029588 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:44.029564 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ceea576-7104-4b64-a0c2-a68ba70cabb8-metrics-certs\") pod \"maas-controller-b67978596-6r6f9\" (UID: \"2ceea576-7104-4b64-a0c2-a68ba70cabb8\") " pod="opendatahub/maas-controller-b67978596-6r6f9" Nov 03 00:21:44.029659 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:44.029592 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ceea576-7104-4b64-a0c2-a68ba70cabb8-webhook-cert\") pod \"maas-controller-b67978596-6r6f9\" (UID: \"2ceea576-7104-4b64-a0c2-a68ba70cabb8\") " pod="opendatahub/maas-controller-b67978596-6r6f9" Nov 03 00:21:44.062750 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:44.062674 5076 generic.go:356] "Generic (PLEG): container finished" podID="1739f335-a175-4d60-952d-5502f4eb2fb1" containerID="08cb1b05672b13d0177379c810b515990102386a263476cd02407b04be9e8143" exitCode=0 Nov 03 00:21:44.062750 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:44.062712 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" event={"ID":"1739f335-a175-4d60-952d-5502f4eb2fb1","Type":"ContainerDied","Data":"08cb1b05672b13d0177379c810b515990102386a263476cd02407b04be9e8143"} Nov 03 00:21:44.062750 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:44.062726 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" Nov 03 00:21:44.063012 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:44.062754 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s" event={"ID":"1739f335-a175-4d60-952d-5502f4eb2fb1","Type":"ContainerDied","Data":"35b99a9e3677028855f95b54bafb0d293e328164511b16d83a5103ef8db64653"} Nov 03 00:21:44.063012 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:44.062775 5076 scope.go:122] "RemoveContainer" containerID="08cb1b05672b13d0177379c810b515990102386a263476cd02407b04be9e8143" Nov 03 00:21:44.072366 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:44.072347 5076 scope.go:122] "RemoveContainer" containerID="d4c0ca1e19a494b7f79ee033f5b540a8652d786008ddc1895dc438450206e8fb" Nov 03 00:21:44.080487 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:44.080470 5076 scope.go:122] "RemoveContainer" containerID="08cb1b05672b13d0177379c810b515990102386a263476cd02407b04be9e8143" Nov 03 00:21:44.080733 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:21:44.080711 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08cb1b05672b13d0177379c810b515990102386a263476cd02407b04be9e8143\": container with ID starting with 08cb1b05672b13d0177379c810b515990102386a263476cd02407b04be9e8143 not found: ID does not exist" containerID="08cb1b05672b13d0177379c810b515990102386a263476cd02407b04be9e8143" Nov 03 00:21:44.080809 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:44.080746 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08cb1b05672b13d0177379c810b515990102386a263476cd02407b04be9e8143"} err="failed to get container status \"08cb1b05672b13d0177379c810b515990102386a263476cd02407b04be9e8143\": rpc error: code = NotFound desc = could not find container \"08cb1b05672b13d0177379c810b515990102386a263476cd02407b04be9e8143\": container with ID starting with 08cb1b05672b13d0177379c810b515990102386a263476cd02407b04be9e8143 not found: ID does not exist" Nov 03 00:21:44.080809 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:44.080773 5076 scope.go:122] "RemoveContainer" containerID="d4c0ca1e19a494b7f79ee033f5b540a8652d786008ddc1895dc438450206e8fb" Nov 03 00:21:44.081039 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:21:44.081020 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4c0ca1e19a494b7f79ee033f5b540a8652d786008ddc1895dc438450206e8fb\": container with ID starting with d4c0ca1e19a494b7f79ee033f5b540a8652d786008ddc1895dc438450206e8fb not found: ID does not exist" containerID="d4c0ca1e19a494b7f79ee033f5b540a8652d786008ddc1895dc438450206e8fb" Nov 03 00:21:44.081087 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:44.081045 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4c0ca1e19a494b7f79ee033f5b540a8652d786008ddc1895dc438450206e8fb"} err="failed to get container status \"d4c0ca1e19a494b7f79ee033f5b540a8652d786008ddc1895dc438450206e8fb\": rpc error: code = NotFound desc = could not find container \"d4c0ca1e19a494b7f79ee033f5b540a8652d786008ddc1895dc438450206e8fb\": container with ID starting with d4c0ca1e19a494b7f79ee033f5b540a8652d786008ddc1895dc438450206e8fb not found: ID does not exist" Nov 03 00:21:44.081126 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:44.081105 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld8x4\" (UniqueName: \"kubernetes.io/projected/2ceea576-7104-4b64-a0c2-a68ba70cabb8-kube-api-access-ld8x4\") pod \"maas-controller-b67978596-6r6f9\" (UID: \"2ceea576-7104-4b64-a0c2-a68ba70cabb8\") " pod="opendatahub/maas-controller-b67978596-6r6f9" Nov 03 00:21:44.113024 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:44.112998 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b67978596-6r6f9" Nov 03 00:21:44.222920 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:44.222871 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s"] Nov 03 00:21:44.227123 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:44.227096 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a2a3480b-openshift-default-5df85b7855-txw8s"] Nov 03 00:21:44.479392 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:44.479366 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b67978596-6r6f9"] Nov 03 00:21:44.481761 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:21:44.481735 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ceea576_7104_4b64_a0c2_a68ba70cabb8.slice/crio-ec020c7f74289dc45a966261a09989c19a3487644393a8bf6fa5e364b5f5f5a1 WatchSource:0}: Error finding container ec020c7f74289dc45a966261a09989c19a3487644393a8bf6fa5e364b5f5f5a1: Status 404 returned error can't find the container with id ec020c7f74289dc45a966261a09989c19a3487644393a8bf6fa5e364b5f5f5a1 Nov 03 00:21:45.069946 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:45.069909 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b67978596-6r6f9" event={"ID":"2ceea576-7104-4b64-a0c2-a68ba70cabb8","Type":"ContainerStarted","Data":"ec020c7f74289dc45a966261a09989c19a3487644393a8bf6fa5e364b5f5f5a1"} Nov 03 00:21:45.149146 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:45.149115 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1739f335-a175-4d60-952d-5502f4eb2fb1" path="/var/lib/kubelet/pods/1739f335-a175-4d60-952d-5502f4eb2fb1/volumes" Nov 03 00:21:48.087871 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:48.087838 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b67978596-6r6f9" event={"ID":"2ceea576-7104-4b64-a0c2-a68ba70cabb8","Type":"ContainerStarted","Data":"04138e78b3f3defa260b44463d1d472da42e5e4f9944eb493f79ecd3e2175b8f"} Nov 03 00:21:48.088243 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:48.087924 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-b67978596-6r6f9" Nov 03 00:21:48.200113 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:48.200055 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-b67978596-6r6f9" podStartSLOduration=1.770983926 podStartE2EDuration="5.200040948s" totalImagesPullingTime="3.429057022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:21:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:21:48.198143266 +0000 UTC m=+5357.166675050" watchObservedRunningTime="2026-11-03 00:21:48.200040948 +0000 UTC m=+5357.168572759" Nov 03 00:21:59.095355 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:21:59.095327 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-b67978596-6r6f9" Nov 03 00:22:03.872853 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:03.872817 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb"] Nov 03 00:22:03.873349 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:03.873269 5076 cpu_manager.go:414] "containerMap: removing container" podUID="1739f335-a175-4d60-952d-5502f4eb2fb1" containerName="istio-proxy" Nov 03 00:22:03.873349 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:03.873285 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1739f335-a175-4d60-952d-5502f4eb2fb1" containerName="istio-proxy" Nov 03 00:22:03.877358 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:03.877340 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" Nov 03 00:22:03.908730 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:03.908703 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2s9q\" (UniqueName: \"kubernetes.io/projected/d6db6fe8-b76c-4878-b0f3-bd347b2d0cde-kube-api-access-b2s9q\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gctbb\" (UID: \"d6db6fe8-b76c-4878-b0f3-bd347b2d0cde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" Nov 03 00:22:03.908852 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:03.908744 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6db6fe8-b76c-4878-b0f3-bd347b2d0cde-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gctbb\" (UID: \"d6db6fe8-b76c-4878-b0f3-bd347b2d0cde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" Nov 03 00:22:03.908852 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:03.908767 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6db6fe8-b76c-4878-b0f3-bd347b2d0cde-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gctbb\" (UID: \"d6db6fe8-b76c-4878-b0f3-bd347b2d0cde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" Nov 03 00:22:03.908962 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:03.908862 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6db6fe8-b76c-4878-b0f3-bd347b2d0cde-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gctbb\" (UID: \"d6db6fe8-b76c-4878-b0f3-bd347b2d0cde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" Nov 03 00:22:03.909003 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:03.908955 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6db6fe8-b76c-4878-b0f3-bd347b2d0cde-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gctbb\" (UID: \"d6db6fe8-b76c-4878-b0f3-bd347b2d0cde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" Nov 03 00:22:03.909043 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:03.909005 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6db6fe8-b76c-4878-b0f3-bd347b2d0cde-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gctbb\" (UID: \"d6db6fe8-b76c-4878-b0f3-bd347b2d0cde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" Nov 03 00:22:03.909043 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:03.909025 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6db6fe8-b76c-4878-b0f3-bd347b2d0cde-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gctbb\" (UID: \"d6db6fe8-b76c-4878-b0f3-bd347b2d0cde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" Nov 03 00:22:03.909122 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:03.909102 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6db6fe8-b76c-4878-b0f3-bd347b2d0cde-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gctbb\" (UID: \"d6db6fe8-b76c-4878-b0f3-bd347b2d0cde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" Nov 03 00:22:03.909159 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:03.909146 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6db6fe8-b76c-4878-b0f3-bd347b2d0cde-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gctbb\" (UID: \"d6db6fe8-b76c-4878-b0f3-bd347b2d0cde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" Nov 03 00:22:03.920253 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:03.920223 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq"] Nov 03 00:22:03.923750 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:03.923730 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" Nov 03 00:22:03.933645 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:03.933622 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-tbbgz\"" Nov 03 00:22:03.971135 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:03.971100 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb"] Nov 03 00:22:03.977295 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:03.977265 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq"] Nov 03 00:22:04.010129 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.010096 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6db6fe8-b76c-4878-b0f3-bd347b2d0cde-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gctbb\" (UID: \"d6db6fe8-b76c-4878-b0f3-bd347b2d0cde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" Nov 03 00:22:04.010129 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.010128 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6db6fe8-b76c-4878-b0f3-bd347b2d0cde-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gctbb\" (UID: \"d6db6fe8-b76c-4878-b0f3-bd347b2d0cde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" Nov 03 00:22:04.010374 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.010148 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/506655af-c50a-4207-906e-9f705fc8f9f6-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-784vq\" (UID: \"506655af-c50a-4207-906e-9f705fc8f9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" Nov 03 00:22:04.010374 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.010169 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6db6fe8-b76c-4878-b0f3-bd347b2d0cde-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gctbb\" (UID: \"d6db6fe8-b76c-4878-b0f3-bd347b2d0cde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" Nov 03 00:22:04.010374 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.010188 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/506655af-c50a-4207-906e-9f705fc8f9f6-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-784vq\" (UID: \"506655af-c50a-4207-906e-9f705fc8f9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" Nov 03 00:22:04.010374 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.010296 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/506655af-c50a-4207-906e-9f705fc8f9f6-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-784vq\" (UID: \"506655af-c50a-4207-906e-9f705fc8f9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" Nov 03 00:22:04.010606 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.010390 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/506655af-c50a-4207-906e-9f705fc8f9f6-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-784vq\" (UID: \"506655af-c50a-4207-906e-9f705fc8f9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" Nov 03 00:22:04.010606 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.010431 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6db6fe8-b76c-4878-b0f3-bd347b2d0cde-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gctbb\" (UID: \"d6db6fe8-b76c-4878-b0f3-bd347b2d0cde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" Nov 03 00:22:04.010606 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.010545 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/506655af-c50a-4207-906e-9f705fc8f9f6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-784vq\" (UID: \"506655af-c50a-4207-906e-9f705fc8f9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" Nov 03 00:22:04.010606 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.010595 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2s9q\" (UniqueName: \"kubernetes.io/projected/d6db6fe8-b76c-4878-b0f3-bd347b2d0cde-kube-api-access-b2s9q\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gctbb\" (UID: \"d6db6fe8-b76c-4878-b0f3-bd347b2d0cde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" Nov 03 00:22:04.010834 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.010618 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6db6fe8-b76c-4878-b0f3-bd347b2d0cde-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gctbb\" (UID: \"d6db6fe8-b76c-4878-b0f3-bd347b2d0cde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" Nov 03 00:22:04.010834 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.010628 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6db6fe8-b76c-4878-b0f3-bd347b2d0cde-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gctbb\" (UID: \"d6db6fe8-b76c-4878-b0f3-bd347b2d0cde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" Nov 03 00:22:04.010834 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.010662 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/506655af-c50a-4207-906e-9f705fc8f9f6-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-784vq\" (UID: \"506655af-c50a-4207-906e-9f705fc8f9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" Nov 03 00:22:04.010834 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.010723 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6db6fe8-b76c-4878-b0f3-bd347b2d0cde-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gctbb\" (UID: \"d6db6fe8-b76c-4878-b0f3-bd347b2d0cde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" Nov 03 00:22:04.010834 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.010759 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/506655af-c50a-4207-906e-9f705fc8f9f6-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-784vq\" (UID: \"506655af-c50a-4207-906e-9f705fc8f9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" Nov 03 00:22:04.010834 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.010788 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6db6fe8-b76c-4878-b0f3-bd347b2d0cde-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gctbb\" (UID: \"d6db6fe8-b76c-4878-b0f3-bd347b2d0cde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" Nov 03 00:22:04.010834 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.010823 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/506655af-c50a-4207-906e-9f705fc8f9f6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-784vq\" (UID: \"506655af-c50a-4207-906e-9f705fc8f9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" Nov 03 00:22:04.011224 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.010857 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz4xh\" (UniqueName: \"kubernetes.io/projected/506655af-c50a-4207-906e-9f705fc8f9f6-kube-api-access-jz4xh\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-784vq\" (UID: \"506655af-c50a-4207-906e-9f705fc8f9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" Nov 03 00:22:04.011224 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.010907 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6db6fe8-b76c-4878-b0f3-bd347b2d0cde-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gctbb\" (UID: \"d6db6fe8-b76c-4878-b0f3-bd347b2d0cde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" Nov 03 00:22:04.011224 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.010986 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6db6fe8-b76c-4878-b0f3-bd347b2d0cde-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gctbb\" (UID: \"d6db6fe8-b76c-4878-b0f3-bd347b2d0cde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" Nov 03 00:22:04.011224 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.011064 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6db6fe8-b76c-4878-b0f3-bd347b2d0cde-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gctbb\" (UID: \"d6db6fe8-b76c-4878-b0f3-bd347b2d0cde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" Nov 03 00:22:04.011224 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.011090 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6db6fe8-b76c-4878-b0f3-bd347b2d0cde-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gctbb\" (UID: \"d6db6fe8-b76c-4878-b0f3-bd347b2d0cde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" Nov 03 00:22:04.011678 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.011656 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6db6fe8-b76c-4878-b0f3-bd347b2d0cde-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gctbb\" (UID: \"d6db6fe8-b76c-4878-b0f3-bd347b2d0cde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" Nov 03 00:22:04.012929 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.012904 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6db6fe8-b76c-4878-b0f3-bd347b2d0cde-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gctbb\" (UID: \"d6db6fe8-b76c-4878-b0f3-bd347b2d0cde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" Nov 03 00:22:04.013250 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.013234 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6db6fe8-b76c-4878-b0f3-bd347b2d0cde-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gctbb\" (UID: \"d6db6fe8-b76c-4878-b0f3-bd347b2d0cde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" Nov 03 00:22:04.064120 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.064090 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6db6fe8-b76c-4878-b0f3-bd347b2d0cde-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gctbb\" (UID: \"d6db6fe8-b76c-4878-b0f3-bd347b2d0cde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" Nov 03 00:22:04.075656 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.075632 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2s9q\" (UniqueName: \"kubernetes.io/projected/d6db6fe8-b76c-4878-b0f3-bd347b2d0cde-kube-api-access-b2s9q\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gctbb\" (UID: \"d6db6fe8-b76c-4878-b0f3-bd347b2d0cde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" Nov 03 00:22:04.112586 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.112550 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/506655af-c50a-4207-906e-9f705fc8f9f6-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-784vq\" (UID: \"506655af-c50a-4207-906e-9f705fc8f9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" Nov 03 00:22:04.112586 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.112589 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/506655af-c50a-4207-906e-9f705fc8f9f6-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-784vq\" (UID: \"506655af-c50a-4207-906e-9f705fc8f9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" Nov 03 00:22:04.112835 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.112610 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/506655af-c50a-4207-906e-9f705fc8f9f6-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-784vq\" (UID: \"506655af-c50a-4207-906e-9f705fc8f9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" Nov 03 00:22:04.112835 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.112647 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/506655af-c50a-4207-906e-9f705fc8f9f6-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-784vq\" (UID: \"506655af-c50a-4207-906e-9f705fc8f9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" Nov 03 00:22:04.112835 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.112678 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/506655af-c50a-4207-906e-9f705fc8f9f6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-784vq\" (UID: \"506655af-c50a-4207-906e-9f705fc8f9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" Nov 03 00:22:04.112835 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.112720 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/506655af-c50a-4207-906e-9f705fc8f9f6-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-784vq\" (UID: \"506655af-c50a-4207-906e-9f705fc8f9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" Nov 03 00:22:04.112835 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.112808 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/506655af-c50a-4207-906e-9f705fc8f9f6-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-784vq\" (UID: \"506655af-c50a-4207-906e-9f705fc8f9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" Nov 03 00:22:04.112835 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.112834 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/506655af-c50a-4207-906e-9f705fc8f9f6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-784vq\" (UID: \"506655af-c50a-4207-906e-9f705fc8f9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" Nov 03 00:22:04.113215 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.112859 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz4xh\" (UniqueName: \"kubernetes.io/projected/506655af-c50a-4207-906e-9f705fc8f9f6-kube-api-access-jz4xh\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-784vq\" (UID: \"506655af-c50a-4207-906e-9f705fc8f9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" Nov 03 00:22:04.113215 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.113040 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/506655af-c50a-4207-906e-9f705fc8f9f6-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-784vq\" (UID: \"506655af-c50a-4207-906e-9f705fc8f9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" Nov 03 00:22:04.113215 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.113106 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/506655af-c50a-4207-906e-9f705fc8f9f6-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-784vq\" (UID: \"506655af-c50a-4207-906e-9f705fc8f9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" Nov 03 00:22:04.113389 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.113305 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/506655af-c50a-4207-906e-9f705fc8f9f6-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-784vq\" (UID: \"506655af-c50a-4207-906e-9f705fc8f9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" Nov 03 00:22:04.113452 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.113411 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/506655af-c50a-4207-906e-9f705fc8f9f6-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-784vq\" (UID: \"506655af-c50a-4207-906e-9f705fc8f9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" Nov 03 00:22:04.113734 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.113711 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/506655af-c50a-4207-906e-9f705fc8f9f6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-784vq\" (UID: \"506655af-c50a-4207-906e-9f705fc8f9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" Nov 03 00:22:04.115118 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.115100 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/506655af-c50a-4207-906e-9f705fc8f9f6-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-784vq\" (UID: \"506655af-c50a-4207-906e-9f705fc8f9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" Nov 03 00:22:04.115221 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.115177 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/506655af-c50a-4207-906e-9f705fc8f9f6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-784vq\" (UID: \"506655af-c50a-4207-906e-9f705fc8f9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" Nov 03 00:22:04.167512 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.167442 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/506655af-c50a-4207-906e-9f705fc8f9f6-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-784vq\" (UID: \"506655af-c50a-4207-906e-9f705fc8f9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" Nov 03 00:22:04.167648 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.167584 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz4xh\" (UniqueName: \"kubernetes.io/projected/506655af-c50a-4207-906e-9f705fc8f9f6-kube-api-access-jz4xh\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-784vq\" (UID: \"506655af-c50a-4207-906e-9f705fc8f9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" Nov 03 00:22:04.190627 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.190590 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" Nov 03 00:22:04.236246 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.236215 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" Nov 03 00:22:04.609735 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.609650 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb"] Nov 03 00:22:04.611166 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:22:04.611140 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6db6fe8_b76c_4878_b0f3_bd347b2d0cde.slice/crio-ab281eea3271bd66baf78839cc3563d7a023b76dc7c7844b14ff7e232185ed5c WatchSource:0}: Error finding container ab281eea3271bd66baf78839cc3563d7a023b76dc7c7844b14ff7e232185ed5c: Status 404 returned error can't find the container with id ab281eea3271bd66baf78839cc3563d7a023b76dc7c7844b14ff7e232185ed5c Nov 03 00:22:04.611432 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.611409 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq"] Nov 03 00:22:04.613467 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.613434 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:22:04.613575 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.613526 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:22:04.613645 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.613576 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:22:04.649051 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:22:04.649024 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod506655af_c50a_4207_906e_9f705fc8f9f6.slice/crio-fcc01e12f0a5a3dc8da1d383727983790890376432d438c9751ffa9d09824490 WatchSource:0}: Error finding container fcc01e12f0a5a3dc8da1d383727983790890376432d438c9751ffa9d09824490: Status 404 returned error can't find the container with id fcc01e12f0a5a3dc8da1d383727983790890376432d438c9751ffa9d09824490 Nov 03 00:22:04.651991 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.651956 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:22:04.652080 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.652045 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:22:04.652129 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:04.652084 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:22:05.170658 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:05.170623 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" event={"ID":"506655af-c50a-4207-906e-9f705fc8f9f6","Type":"ContainerStarted","Data":"5a1a95f9d343fc9ac449ab8af8f919e186f966450729d2ad615e6a5afc94c95e"} Nov 03 00:22:05.171122 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:05.170679 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" event={"ID":"506655af-c50a-4207-906e-9f705fc8f9f6","Type":"ContainerStarted","Data":"fcc01e12f0a5a3dc8da1d383727983790890376432d438c9751ffa9d09824490"} Nov 03 00:22:05.172254 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:05.172233 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" event={"ID":"d6db6fe8-b76c-4878-b0f3-bd347b2d0cde","Type":"ContainerStarted","Data":"f3ae953d3ca4f15694d1f60751ad945c41054e7839db78764c8a99c64310f152"} Nov 03 00:22:05.172319 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:05.172277 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" event={"ID":"d6db6fe8-b76c-4878-b0f3-bd347b2d0cde","Type":"ContainerStarted","Data":"ab281eea3271bd66baf78839cc3563d7a023b76dc7c7844b14ff7e232185ed5c"} Nov 03 00:22:05.190837 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:05.190816 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" Nov 03 00:22:05.192305 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:05.192282 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gctbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 00:22:05.192417 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:05.192323 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" podUID="d6db6fe8-b76c-4878-b0f3-bd347b2d0cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 00:22:05.238912 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:05.238261 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" Nov 03 00:22:05.242911 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:05.240317 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-784vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Nov 03 00:22:05.242911 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:05.240375 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" podUID="506655af-c50a-4207-906e-9f705fc8f9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Nov 03 00:22:05.289399 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:05.289341 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" podStartSLOduration=2.289327162 podStartE2EDuration="2.289327162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:22:05.288054158 +0000 UTC m=+5374.256585940" watchObservedRunningTime="2026-11-03 00:22:05.289327162 +0000 UTC m=+5374.257858947" Nov 03 00:22:05.424972 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:05.421624 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" podStartSLOduration=2.421603848 podStartE2EDuration="2.421603848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:22:05.418451298 +0000 UTC m=+5374.386983083" watchObservedRunningTime="2026-11-03 00:22:05.421603848 +0000 UTC m=+5374.390135633" Nov 03 00:22:06.191228 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:06.191197 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gctbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 00:22:06.191651 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:06.191246 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" podUID="d6db6fe8-b76c-4878-b0f3-bd347b2d0cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 00:22:06.240913 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:06.240282 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-784vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Nov 03 00:22:06.240913 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:06.240342 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" podUID="506655af-c50a-4207-906e-9f705fc8f9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Nov 03 00:22:07.190954 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:07.190915 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gctbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 00:22:07.191180 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:07.190958 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" podUID="d6db6fe8-b76c-4878-b0f3-bd347b2d0cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 00:22:07.266261 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:07.243917 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-784vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Nov 03 00:22:07.266261 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:07.243985 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" podUID="506655af-c50a-4207-906e-9f705fc8f9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Nov 03 00:22:08.191947 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:08.191911 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gctbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 00:22:08.192156 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:08.191969 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" podUID="d6db6fe8-b76c-4878-b0f3-bd347b2d0cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 00:22:08.237979 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:08.237939 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-784vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Nov 03 00:22:08.238238 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:08.238210 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" podUID="506655af-c50a-4207-906e-9f705fc8f9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Nov 03 00:22:08.401125 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:08.401088 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-n5x8b"] Nov 03 00:22:08.404002 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:08.403983 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-n5x8b" Nov 03 00:22:08.461968 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:08.458554 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-n5x8b"] Nov 03 00:22:08.462841 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:08.462816 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:22:08.487528 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:08.487495 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d029e6d-414e-4328-ad80-635de1c9bb35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-n5x8b\" (UID: \"3d029e6d-414e-4328-ad80-635de1c9bb35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-n5x8b" Nov 03 00:22:08.487829 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:08.487815 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkw4m\" (UniqueName: \"kubernetes.io/projected/3d029e6d-414e-4328-ad80-635de1c9bb35-kube-api-access-dkw4m\") pod \"maas-api-revoke-sub-85237e45-n5x8b\" (UID: \"3d029e6d-414e-4328-ad80-635de1c9bb35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-n5x8b" Nov 03 00:22:08.487990 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:08.487975 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d029e6d-414e-4328-ad80-635de1c9bb35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-n5x8b\" (UID: \"3d029e6d-414e-4328-ad80-635de1c9bb35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-n5x8b" Nov 03 00:22:08.502514 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:08.502494 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:22:08.589637 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:08.589597 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkw4m\" (UniqueName: \"kubernetes.io/projected/3d029e6d-414e-4328-ad80-635de1c9bb35-kube-api-access-dkw4m\") pod \"maas-api-revoke-sub-85237e45-n5x8b\" (UID: \"3d029e6d-414e-4328-ad80-635de1c9bb35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-n5x8b" Nov 03 00:22:08.589819 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:08.589644 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d029e6d-414e-4328-ad80-635de1c9bb35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-n5x8b\" (UID: \"3d029e6d-414e-4328-ad80-635de1c9bb35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-n5x8b" Nov 03 00:22:08.589819 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:08.589763 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d029e6d-414e-4328-ad80-635de1c9bb35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-n5x8b\" (UID: \"3d029e6d-414e-4328-ad80-635de1c9bb35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-n5x8b" Nov 03 00:22:08.590539 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:08.590512 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d029e6d-414e-4328-ad80-635de1c9bb35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-n5x8b\" (UID: \"3d029e6d-414e-4328-ad80-635de1c9bb35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-n5x8b" Nov 03 00:22:08.592170 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:08.592144 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d029e6d-414e-4328-ad80-635de1c9bb35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-n5x8b\" (UID: \"3d029e6d-414e-4328-ad80-635de1c9bb35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-n5x8b" Nov 03 00:22:08.658708 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:08.658643 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkw4m\" (UniqueName: \"kubernetes.io/projected/3d029e6d-414e-4328-ad80-635de1c9bb35-kube-api-access-dkw4m\") pod \"maas-api-revoke-sub-85237e45-n5x8b\" (UID: \"3d029e6d-414e-4328-ad80-635de1c9bb35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-n5x8b" Nov 03 00:22:08.732149 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:08.732056 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-n5x8b" Nov 03 00:22:09.179372 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:09.179349 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-n5x8b"] Nov 03 00:22:09.181931 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:22:09.181905 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d029e6d_414e_4328_ad80_635de1c9bb35.slice/crio-61d664a0f96d315d3bf103c8e7c1c1b730dee2ef2602d67b3133d24ded9fb3f1 WatchSource:0}: Error finding container 61d664a0f96d315d3bf103c8e7c1c1b730dee2ef2602d67b3133d24ded9fb3f1: Status 404 returned error can't find the container with id 61d664a0f96d315d3bf103c8e7c1c1b730dee2ef2602d67b3133d24ded9fb3f1 Nov 03 00:22:09.191238 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:09.191207 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gctbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 00:22:09.191398 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:09.191265 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" podUID="d6db6fe8-b76c-4878-b0f3-bd347b2d0cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 00:22:09.194296 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:09.194272 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-n5x8b" event={"ID":"3d029e6d-414e-4328-ad80-635de1c9bb35","Type":"ContainerStarted","Data":"61d664a0f96d315d3bf103c8e7c1c1b730dee2ef2602d67b3133d24ded9fb3f1"} Nov 03 00:22:09.251924 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:09.251311 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-784vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Nov 03 00:22:09.251924 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:09.251367 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" podUID="506655af-c50a-4207-906e-9f705fc8f9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Nov 03 00:22:09.888737 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:22:09.888658 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:22:10.191535 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:10.191497 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gctbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 00:22:10.191714 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:10.191557 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" podUID="d6db6fe8-b76c-4878-b0f3-bd347b2d0cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 00:22:10.201090 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:10.201061 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-n5x8b" event={"ID":"3d029e6d-414e-4328-ad80-635de1c9bb35","Type":"ContainerStarted","Data":"92e5e0fc235f935d97bfa6d166ac9608b0560b16d88f2df1a26da6fa389da219"} Nov 03 00:22:10.240912 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:10.240287 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-784vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Nov 03 00:22:10.240912 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:10.240350 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" podUID="506655af-c50a-4207-906e-9f705fc8f9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Nov 03 00:22:10.342913 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:10.342603 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-n5x8b" podStartSLOduration=2.342585093 podStartE2EDuration="2.342585093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:22:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:22:10.342420017 +0000 UTC m=+5379.310951804" watchObservedRunningTime="2026-11-03 00:22:10.342585093 +0000 UTC m=+5379.311116879" Nov 03 00:22:11.191436 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:11.191397 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gctbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 00:22:11.191941 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:11.191452 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" podUID="d6db6fe8-b76c-4878-b0f3-bd347b2d0cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 00:22:11.211654 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:11.211615 5076 generic.go:356] "Generic (PLEG): container finished" podID="3d029e6d-414e-4328-ad80-635de1c9bb35" containerID="92e5e0fc235f935d97bfa6d166ac9608b0560b16d88f2df1a26da6fa389da219" exitCode=0 Nov 03 00:22:11.211834 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:11.211682 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-n5x8b" event={"ID":"3d029e6d-414e-4328-ad80-635de1c9bb35","Type":"ContainerDied","Data":"92e5e0fc235f935d97bfa6d166ac9608b0560b16d88f2df1a26da6fa389da219"} Nov 03 00:22:11.237239 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:11.237199 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-784vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Nov 03 00:22:11.237408 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:11.237267 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" podUID="506655af-c50a-4207-906e-9f705fc8f9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Nov 03 00:22:12.191393 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:12.191355 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gctbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 00:22:12.191567 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:12.191424 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" podUID="d6db6fe8-b76c-4878-b0f3-bd347b2d0cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 00:22:12.236977 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:12.236940 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-784vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Nov 03 00:22:12.237142 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:12.237006 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" podUID="506655af-c50a-4207-906e-9f705fc8f9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Nov 03 00:22:12.395207 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:12.395169 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-n5x8b" Nov 03 00:22:12.430115 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:12.430077 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d029e6d-414e-4328-ad80-635de1c9bb35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d029e6d-414e-4328-ad80-635de1c9bb35-maas-api-xks-ca\") pod \"3d029e6d-414e-4328-ad80-635de1c9bb35\" (UID: \"3d029e6d-414e-4328-ad80-635de1c9bb35\") " Nov 03 00:22:12.430320 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:12.430229 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d029e6d-414e-4328-ad80-635de1c9bb35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d029e6d-414e-4328-ad80-635de1c9bb35-maas-api-openshift-ca\") pod \"3d029e6d-414e-4328-ad80-635de1c9bb35\" (UID: \"3d029e6d-414e-4328-ad80-635de1c9bb35\") " Nov 03 00:22:12.430320 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:12.430280 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d029e6d-414e-4328-ad80-635de1c9bb35-kube-api-access-dkw4m\" (UniqueName: \"kubernetes.io/projected/3d029e6d-414e-4328-ad80-635de1c9bb35-kube-api-access-dkw4m\") pod \"3d029e6d-414e-4328-ad80-635de1c9bb35\" (UID: \"3d029e6d-414e-4328-ad80-635de1c9bb35\") " Nov 03 00:22:12.430923 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:12.430874 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d029e6d-414e-4328-ad80-635de1c9bb35-maas-api-openshift-ca" pod "3d029e6d-414e-4328-ad80-635de1c9bb35" (UID: "3d029e6d-414e-4328-ad80-635de1c9bb35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:22:12.439060 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:12.439015 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d029e6d-414e-4328-ad80-635de1c9bb35-maas-api-xks-ca" pod "3d029e6d-414e-4328-ad80-635de1c9bb35" (UID: "3d029e6d-414e-4328-ad80-635de1c9bb35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:22:12.439192 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:12.439072 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d029e6d-414e-4328-ad80-635de1c9bb35-kube-api-access-dkw4m" pod "3d029e6d-414e-4328-ad80-635de1c9bb35" (UID: "3d029e6d-414e-4328-ad80-635de1c9bb35"). InnerVolumeSpecName "kube-api-access-dkw4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:22:12.534836 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:12.533865 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d029e6d-414e-4328-ad80-635de1c9bb35-maas-api-openshift-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:22:12.534836 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:12.533929 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkw4m\" (UniqueName: \"kubernetes.io/projected/3d029e6d-414e-4328-ad80-635de1c9bb35-kube-api-access-dkw4m\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:22:12.534836 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:12.533948 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d029e6d-414e-4328-ad80-635de1c9bb35-maas-api-xks-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:22:13.225555 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:13.225521 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-n5x8b" Nov 03 00:22:13.226006 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:13.225545 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-n5x8b" event={"ID":"3d029e6d-414e-4328-ad80-635de1c9bb35","Type":"ContainerDied","Data":"61d664a0f96d315d3bf103c8e7c1c1b730dee2ef2602d67b3133d24ded9fb3f1"} Nov 03 00:22:13.226006 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:13.225604 5076 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61d664a0f96d315d3bf103c8e7c1c1b730dee2ef2602d67b3133d24ded9fb3f1" Nov 03 00:22:14.231746 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:14.231707 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gctbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:22:14.232702 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:14.232668 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" podUID="d6db6fe8-b76c-4878-b0f3-bd347b2d0cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:22:14.278024 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:14.277993 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-784vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:22:14.278207 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:14.278041 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" podUID="506655af-c50a-4207-906e-9f705fc8f9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:22:15.273394 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:15.273364 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gctbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:22:15.273823 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:15.273654 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" podUID="d6db6fe8-b76c-4878-b0f3-bd347b2d0cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:22:15.319097 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:15.319059 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-784vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:22:15.319335 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:15.319125 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" podUID="506655af-c50a-4207-906e-9f705fc8f9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:22:15.413682 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:15.413647 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" Nov 03 00:22:15.414319 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:15.414294 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" Nov 03 00:22:15.533315 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:15.533226 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" Nov 03 00:22:15.766353 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:15.766326 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" Nov 03 00:22:15.766650 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:15.766619 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" Nov 03 00:22:15.767522 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:15.767500 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" Nov 03 00:22:20.646587 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:20.646550 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b67978596-6r6f9"] Nov 03 00:22:20.647212 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:20.646872 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-b67978596-6r6f9" podUID="2ceea576-7104-4b64-a0c2-a68ba70cabb8" containerName="manager" containerID="cri-o://04138e78b3f3defa260b44463d1d472da42e5e4f9944eb493f79ecd3e2175b8f" gracePeriod=30 Nov 03 00:22:20.910478 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:20.910421 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b67978596-6r6f9" Nov 03 00:22:21.029776 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:21.029742 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ceea576-7104-4b64-a0c2-a68ba70cabb8-kube-api-access-ld8x4\" (UniqueName: \"kubernetes.io/projected/2ceea576-7104-4b64-a0c2-a68ba70cabb8-kube-api-access-ld8x4\") pod \"2ceea576-7104-4b64-a0c2-a68ba70cabb8\" (UID: \"2ceea576-7104-4b64-a0c2-a68ba70cabb8\") " Nov 03 00:22:21.030179 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:21.030148 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ceea576-7104-4b64-a0c2-a68ba70cabb8-webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ceea576-7104-4b64-a0c2-a68ba70cabb8-webhook-cert\") pod \"2ceea576-7104-4b64-a0c2-a68ba70cabb8\" (UID: \"2ceea576-7104-4b64-a0c2-a68ba70cabb8\") " Nov 03 00:22:21.030679 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:21.030653 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ceea576-7104-4b64-a0c2-a68ba70cabb8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ceea576-7104-4b64-a0c2-a68ba70cabb8-metrics-certs\") pod \"2ceea576-7104-4b64-a0c2-a68ba70cabb8\" (UID: \"2ceea576-7104-4b64-a0c2-a68ba70cabb8\") " Nov 03 00:22:21.032969 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:21.032938 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ceea576-7104-4b64-a0c2-a68ba70cabb8-kube-api-access-ld8x4" pod "2ceea576-7104-4b64-a0c2-a68ba70cabb8" (UID: "2ceea576-7104-4b64-a0c2-a68ba70cabb8"). InnerVolumeSpecName "kube-api-access-ld8x4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:22:21.033074 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:21.033019 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ceea576-7104-4b64-a0c2-a68ba70cabb8-webhook-cert" pod "2ceea576-7104-4b64-a0c2-a68ba70cabb8" (UID: "2ceea576-7104-4b64-a0c2-a68ba70cabb8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:22:21.034016 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:21.033994 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ceea576-7104-4b64-a0c2-a68ba70cabb8-metrics-certs" pod "2ceea576-7104-4b64-a0c2-a68ba70cabb8" (UID: "2ceea576-7104-4b64-a0c2-a68ba70cabb8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:22:21.132502 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:21.132460 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ceea576-7104-4b64-a0c2-a68ba70cabb8-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:22:21.132502 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:21.132504 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld8x4\" (UniqueName: \"kubernetes.io/projected/2ceea576-7104-4b64-a0c2-a68ba70cabb8-kube-api-access-ld8x4\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:22:21.132787 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:21.132522 5076 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ceea576-7104-4b64-a0c2-a68ba70cabb8-webhook-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:22:21.278874 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:21.278841 5076 generic.go:356] "Generic (PLEG): container finished" podID="2ceea576-7104-4b64-a0c2-a68ba70cabb8" containerID="04138e78b3f3defa260b44463d1d472da42e5e4f9944eb493f79ecd3e2175b8f" exitCode=0 Nov 03 00:22:21.279070 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:21.278897 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b67978596-6r6f9" Nov 03 00:22:21.279070 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:21.278938 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b67978596-6r6f9" event={"ID":"2ceea576-7104-4b64-a0c2-a68ba70cabb8","Type":"ContainerDied","Data":"04138e78b3f3defa260b44463d1d472da42e5e4f9944eb493f79ecd3e2175b8f"} Nov 03 00:22:21.279070 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:21.278986 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b67978596-6r6f9" event={"ID":"2ceea576-7104-4b64-a0c2-a68ba70cabb8","Type":"ContainerDied","Data":"ec020c7f74289dc45a966261a09989c19a3487644393a8bf6fa5e364b5f5f5a1"} Nov 03 00:22:21.279070 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:21.279007 5076 scope.go:122] "RemoveContainer" containerID="04138e78b3f3defa260b44463d1d472da42e5e4f9944eb493f79ecd3e2175b8f" Nov 03 00:22:21.287778 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:21.287760 5076 scope.go:122] "RemoveContainer" containerID="04138e78b3f3defa260b44463d1d472da42e5e4f9944eb493f79ecd3e2175b8f" Nov 03 00:22:21.288044 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:22:21.288026 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04138e78b3f3defa260b44463d1d472da42e5e4f9944eb493f79ecd3e2175b8f\": container with ID starting with 04138e78b3f3defa260b44463d1d472da42e5e4f9944eb493f79ecd3e2175b8f not found: ID does not exist" containerID="04138e78b3f3defa260b44463d1d472da42e5e4f9944eb493f79ecd3e2175b8f" Nov 03 00:22:21.288109 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:21.288053 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04138e78b3f3defa260b44463d1d472da42e5e4f9944eb493f79ecd3e2175b8f"} err="failed to get container status \"04138e78b3f3defa260b44463d1d472da42e5e4f9944eb493f79ecd3e2175b8f\": rpc error: code = NotFound desc = could not find container \"04138e78b3f3defa260b44463d1d472da42e5e4f9944eb493f79ecd3e2175b8f\": container with ID starting with 04138e78b3f3defa260b44463d1d472da42e5e4f9944eb493f79ecd3e2175b8f not found: ID does not exist" Nov 03 00:22:21.437855 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:21.437823 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b67978596-6r6f9"] Nov 03 00:22:21.485183 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:21.485143 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-b67978596-6r6f9"] Nov 03 00:22:22.771842 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:22.771805 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-b67978596-lth4c"] Nov 03 00:22:22.772289 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:22.772212 5076 cpu_manager.go:414] "containerMap: removing container" podUID="2ceea576-7104-4b64-a0c2-a68ba70cabb8" containerName="manager" Nov 03 00:22:22.772289 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:22.772224 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ceea576-7104-4b64-a0c2-a68ba70cabb8" containerName="manager" Nov 03 00:22:22.772289 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:22.772262 5076 cpu_manager.go:414] "containerMap: removing container" podUID="3d029e6d-414e-4328-ad80-635de1c9bb35" containerName="revoke-keys" Nov 03 00:22:22.772289 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:22.772267 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d029e6d-414e-4328-ad80-635de1c9bb35" containerName="revoke-keys" Nov 03 00:22:22.772446 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:22.772337 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d029e6d-414e-4328-ad80-635de1c9bb35" containerName="revoke-keys" Nov 03 00:22:22.772446 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:22.772345 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ceea576-7104-4b64-a0c2-a68ba70cabb8" containerName="manager" Nov 03 00:22:22.776703 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:22.776683 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b67978596-lth4c" Nov 03 00:22:22.826529 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:22.826502 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 00:22:22.849214 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:22.849187 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtlls\" (UniqueName: \"kubernetes.io/projected/b54d7b4b-cee5-47fc-a662-1b24a9d96356-kube-api-access-mtlls\") pod \"maas-controller-b67978596-lth4c\" (UID: \"b54d7b4b-cee5-47fc-a662-1b24a9d96356\") " pod="opendatahub/maas-controller-b67978596-lth4c" Nov 03 00:22:22.849407 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:22.849237 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b54d7b4b-cee5-47fc-a662-1b24a9d96356-metrics-certs\") pod \"maas-controller-b67978596-lth4c\" (UID: \"b54d7b4b-cee5-47fc-a662-1b24a9d96356\") " pod="opendatahub/maas-controller-b67978596-lth4c" Nov 03 00:22:22.849407 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:22.849357 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b54d7b4b-cee5-47fc-a662-1b24a9d96356-webhook-cert\") pod \"maas-controller-b67978596-lth4c\" (UID: \"b54d7b4b-cee5-47fc-a662-1b24a9d96356\") " pod="opendatahub/maas-controller-b67978596-lth4c" Nov 03 00:22:22.873838 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:22.873804 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8fxqx\"" Nov 03 00:22:22.874019 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:22.873937 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 00:22:22.874080 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:22.874019 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 00:22:22.877293 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:22.877269 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b67978596-lth4c"] Nov 03 00:22:22.950742 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:22.950711 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtlls\" (UniqueName: \"kubernetes.io/projected/b54d7b4b-cee5-47fc-a662-1b24a9d96356-kube-api-access-mtlls\") pod \"maas-controller-b67978596-lth4c\" (UID: \"b54d7b4b-cee5-47fc-a662-1b24a9d96356\") " pod="opendatahub/maas-controller-b67978596-lth4c" Nov 03 00:22:22.950966 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:22.950760 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b54d7b4b-cee5-47fc-a662-1b24a9d96356-metrics-certs\") pod \"maas-controller-b67978596-lth4c\" (UID: \"b54d7b4b-cee5-47fc-a662-1b24a9d96356\") " pod="opendatahub/maas-controller-b67978596-lth4c" Nov 03 00:22:22.950966 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:22.950826 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b54d7b4b-cee5-47fc-a662-1b24a9d96356-webhook-cert\") pod \"maas-controller-b67978596-lth4c\" (UID: \"b54d7b4b-cee5-47fc-a662-1b24a9d96356\") " pod="opendatahub/maas-controller-b67978596-lth4c" Nov 03 00:22:22.953319 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:22.953295 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b54d7b4b-cee5-47fc-a662-1b24a9d96356-metrics-certs\") pod \"maas-controller-b67978596-lth4c\" (UID: \"b54d7b4b-cee5-47fc-a662-1b24a9d96356\") " pod="opendatahub/maas-controller-b67978596-lth4c" Nov 03 00:22:22.953428 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:22.953320 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b54d7b4b-cee5-47fc-a662-1b24a9d96356-webhook-cert\") pod \"maas-controller-b67978596-lth4c\" (UID: \"b54d7b4b-cee5-47fc-a662-1b24a9d96356\") " pod="opendatahub/maas-controller-b67978596-lth4c" Nov 03 00:22:23.005582 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:23.005548 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtlls\" (UniqueName: \"kubernetes.io/projected/b54d7b4b-cee5-47fc-a662-1b24a9d96356-kube-api-access-mtlls\") pod \"maas-controller-b67978596-lth4c\" (UID: \"b54d7b4b-cee5-47fc-a662-1b24a9d96356\") " pod="opendatahub/maas-controller-b67978596-lth4c" Nov 03 00:22:23.089869 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:23.089774 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b67978596-lth4c" Nov 03 00:22:23.153409 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:23.152811 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ceea576-7104-4b64-a0c2-a68ba70cabb8" path="/var/lib/kubelet/pods/2ceea576-7104-4b64-a0c2-a68ba70cabb8/volumes" Nov 03 00:22:23.454272 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:23.454247 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b67978596-lth4c"] Nov 03 00:22:23.455843 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:22:23.455819 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb54d7b4b_cee5_47fc_a662_1b24a9d96356.slice/crio-d7e62f383a905468c27df39d4380ab85de3b78d6ec0149adf536ef3bc474e5e1 WatchSource:0}: Error finding container d7e62f383a905468c27df39d4380ab85de3b78d6ec0149adf536ef3bc474e5e1: Status 404 returned error can't find the container with id d7e62f383a905468c27df39d4380ab85de3b78d6ec0149adf536ef3bc474e5e1 Nov 03 00:22:24.296371 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:24.296332 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b67978596-lth4c" event={"ID":"b54d7b4b-cee5-47fc-a662-1b24a9d96356","Type":"ContainerStarted","Data":"d7e62f383a905468c27df39d4380ab85de3b78d6ec0149adf536ef3bc474e5e1"} Nov 03 00:22:25.302897 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:25.302851 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b67978596-lth4c" event={"ID":"b54d7b4b-cee5-47fc-a662-1b24a9d96356","Type":"ContainerStarted","Data":"2a0bcb85a718bde6f4d8670119ab05caea13ade622377239a02ba40b18853895"} Nov 03 00:22:25.303306 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:25.302930 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-b67978596-lth4c" Nov 03 00:22:25.413836 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:25.413769 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-b67978596-lth4c" podStartSLOduration=2.511546127 podStartE2EDuration="3.413756298s" totalImagesPullingTime="902.210171ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:22:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:22:25.413656241 +0000 UTC m=+5394.382188026" watchObservedRunningTime="2026-11-03 00:22:25.413756298 +0000 UTC m=+5394.382288097" Nov 03 00:22:32.220546 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:32.220511 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 03 00:22:32.317904 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:32.317863 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 03 00:22:36.310282 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:36.310250 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-b67978596-lth4c" Nov 03 00:22:39.883848 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:22:39.883810 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:22:53.181279 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:53.181242 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-6ml8x"] Nov 03 00:22:53.184855 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:53.184836 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-6ml8x" Nov 03 00:22:53.239677 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:53.239641 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-6ml8x"] Nov 03 00:22:53.257318 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:53.257284 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/568a402f-785b-4479-91c5-8ae0814e66c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-6ml8x\" (UID: \"568a402f-785b-4479-91c5-8ae0814e66c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-6ml8x" Nov 03 00:22:53.257483 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:53.257361 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/568a402f-785b-4479-91c5-8ae0814e66c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-6ml8x\" (UID: \"568a402f-785b-4479-91c5-8ae0814e66c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-6ml8x" Nov 03 00:22:53.257529 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:53.257491 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8mdd\" (UniqueName: \"kubernetes.io/projected/568a402f-785b-4479-91c5-8ae0814e66c0-kube-api-access-x8mdd\") pod \"maas-api-revoke-sub-f904bff9-6ml8x\" (UID: \"568a402f-785b-4479-91c5-8ae0814e66c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-6ml8x" Nov 03 00:22:53.283602 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:53.283570 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:22:53.283602 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:53.283583 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:22:53.358187 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:53.358150 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/568a402f-785b-4479-91c5-8ae0814e66c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-6ml8x\" (UID: \"568a402f-785b-4479-91c5-8ae0814e66c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-6ml8x" Nov 03 00:22:53.358360 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:53.358200 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8mdd\" (UniqueName: \"kubernetes.io/projected/568a402f-785b-4479-91c5-8ae0814e66c0-kube-api-access-x8mdd\") pod \"maas-api-revoke-sub-f904bff9-6ml8x\" (UID: \"568a402f-785b-4479-91c5-8ae0814e66c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-6ml8x" Nov 03 00:22:53.358360 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:53.358240 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/568a402f-785b-4479-91c5-8ae0814e66c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-6ml8x\" (UID: \"568a402f-785b-4479-91c5-8ae0814e66c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-6ml8x" Nov 03 00:22:53.358862 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:53.358839 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/568a402f-785b-4479-91c5-8ae0814e66c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-6ml8x\" (UID: \"568a402f-785b-4479-91c5-8ae0814e66c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-6ml8x" Nov 03 00:22:53.360793 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:53.360761 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/568a402f-785b-4479-91c5-8ae0814e66c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-6ml8x\" (UID: \"568a402f-785b-4479-91c5-8ae0814e66c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-6ml8x" Nov 03 00:22:53.414040 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:53.414004 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8mdd\" (UniqueName: \"kubernetes.io/projected/568a402f-785b-4479-91c5-8ae0814e66c0-kube-api-access-x8mdd\") pod \"maas-api-revoke-sub-f904bff9-6ml8x\" (UID: \"568a402f-785b-4479-91c5-8ae0814e66c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-6ml8x" Nov 03 00:22:53.498961 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:53.498926 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-6ml8x" Nov 03 00:22:53.869483 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:53.869455 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-6ml8x"] Nov 03 00:22:53.871135 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:22:53.871104 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod568a402f_785b_4479_91c5_8ae0814e66c0.slice/crio-ff9c0e3bbbe9f85b140e1a6aba5a11e02e9cc38280bea563b69d4499a606aec5 WatchSource:0}: Error finding container ff9c0e3bbbe9f85b140e1a6aba5a11e02e9cc38280bea563b69d4499a606aec5: Status 404 returned error can't find the container with id ff9c0e3bbbe9f85b140e1a6aba5a11e02e9cc38280bea563b69d4499a606aec5 Nov 03 00:22:54.460536 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:54.460493 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-6ml8x" event={"ID":"568a402f-785b-4479-91c5-8ae0814e66c0","Type":"ContainerStarted","Data":"65e60aa59f6eeb37a45a30aec23e7846347a93b0bdeda64ea956d150d3db2c51"} Nov 03 00:22:54.460959 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:54.460577 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-6ml8x" event={"ID":"568a402f-785b-4479-91c5-8ae0814e66c0","Type":"ContainerStarted","Data":"ff9c0e3bbbe9f85b140e1a6aba5a11e02e9cc38280bea563b69d4499a606aec5"} Nov 03 00:22:54.570859 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:54.570802 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-6ml8x" podStartSLOduration=1.570788851 podStartE2EDuration="1.570788851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:22:54.568396186 +0000 UTC m=+5423.536927965" watchObservedRunningTime="2026-11-03 00:22:54.570788851 +0000 UTC m=+5423.539320634" Nov 03 00:22:55.466747 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:55.466710 5076 generic.go:356] "Generic (PLEG): container finished" podID="568a402f-785b-4479-91c5-8ae0814e66c0" containerID="65e60aa59f6eeb37a45a30aec23e7846347a93b0bdeda64ea956d150d3db2c51" exitCode=0 Nov 03 00:22:55.467201 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:55.466797 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-6ml8x" event={"ID":"568a402f-785b-4479-91c5-8ae0814e66c0","Type":"ContainerDied","Data":"65e60aa59f6eeb37a45a30aec23e7846347a93b0bdeda64ea956d150d3db2c51"} Nov 03 00:22:56.612301 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:56.612274 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-6ml8x" Nov 03 00:22:56.689640 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:56.689603 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/568a402f-785b-4479-91c5-8ae0814e66c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/568a402f-785b-4479-91c5-8ae0814e66c0-maas-api-openshift-ca\") pod \"568a402f-785b-4479-91c5-8ae0814e66c0\" (UID: \"568a402f-785b-4479-91c5-8ae0814e66c0\") " Nov 03 00:22:56.689822 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:56.689729 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/568a402f-785b-4479-91c5-8ae0814e66c0-kube-api-access-x8mdd\" (UniqueName: \"kubernetes.io/projected/568a402f-785b-4479-91c5-8ae0814e66c0-kube-api-access-x8mdd\") pod \"568a402f-785b-4479-91c5-8ae0814e66c0\" (UID: \"568a402f-785b-4479-91c5-8ae0814e66c0\") " Nov 03 00:22:56.689822 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:56.689764 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/568a402f-785b-4479-91c5-8ae0814e66c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/568a402f-785b-4479-91c5-8ae0814e66c0-maas-api-xks-ca\") pod \"568a402f-785b-4479-91c5-8ae0814e66c0\" (UID: \"568a402f-785b-4479-91c5-8ae0814e66c0\") " Nov 03 00:22:56.690012 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:56.689987 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/568a402f-785b-4479-91c5-8ae0814e66c0-maas-api-openshift-ca" pod "568a402f-785b-4479-91c5-8ae0814e66c0" (UID: "568a402f-785b-4479-91c5-8ae0814e66c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:22:56.690090 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:56.690075 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/568a402f-785b-4479-91c5-8ae0814e66c0-maas-api-openshift-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:22:56.691811 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:56.691779 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/568a402f-785b-4479-91c5-8ae0814e66c0-maas-api-xks-ca" pod "568a402f-785b-4479-91c5-8ae0814e66c0" (UID: "568a402f-785b-4479-91c5-8ae0814e66c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:22:56.691960 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:56.691864 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/568a402f-785b-4479-91c5-8ae0814e66c0-kube-api-access-x8mdd" pod "568a402f-785b-4479-91c5-8ae0814e66c0" (UID: "568a402f-785b-4479-91c5-8ae0814e66c0"). InnerVolumeSpecName "kube-api-access-x8mdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:22:56.791373 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:56.791292 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/568a402f-785b-4479-91c5-8ae0814e66c0-maas-api-xks-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:22:56.791373 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:56.791330 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8mdd\" (UniqueName: \"kubernetes.io/projected/568a402f-785b-4479-91c5-8ae0814e66c0-kube-api-access-x8mdd\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:22:57.478187 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:57.478157 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-6ml8x" Nov 03 00:22:57.478361 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:57.478156 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-6ml8x" event={"ID":"568a402f-785b-4479-91c5-8ae0814e66c0","Type":"ContainerDied","Data":"ff9c0e3bbbe9f85b140e1a6aba5a11e02e9cc38280bea563b69d4499a606aec5"} Nov 03 00:22:57.478361 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:22:57.478289 5076 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff9c0e3bbbe9f85b140e1a6aba5a11e02e9cc38280bea563b69d4499a606aec5" Nov 03 00:23:09.887839 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:23:09.887794 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:23:18.928462 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:18.928257 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r"] Nov 03 00:23:18.928963 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:18.928939 5076 cpu_manager.go:414] "containerMap: removing container" podUID="568a402f-785b-4479-91c5-8ae0814e66c0" containerName="revoke-keys" Nov 03 00:23:18.928963 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:18.928957 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="568a402f-785b-4479-91c5-8ae0814e66c0" containerName="revoke-keys" Nov 03 00:23:18.929116 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:18.929108 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="568a402f-785b-4479-91c5-8ae0814e66c0" containerName="revoke-keys" Nov 03 00:23:18.932760 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:18.932739 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" Nov 03 00:23:19.014047 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:19.013496 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cd92df4-350d-4fe2-bd41-48b57198a268-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5xx2r\" (UID: \"9cd92df4-350d-4fe2-bd41-48b57198a268\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" Nov 03 00:23:19.014047 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:19.013548 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cd92df4-350d-4fe2-bd41-48b57198a268-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5xx2r\" (UID: \"9cd92df4-350d-4fe2-bd41-48b57198a268\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" Nov 03 00:23:19.014047 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:19.013594 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cd92df4-350d-4fe2-bd41-48b57198a268-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5xx2r\" (UID: \"9cd92df4-350d-4fe2-bd41-48b57198a268\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" Nov 03 00:23:19.014047 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:19.013636 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd92df4-350d-4fe2-bd41-48b57198a268-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5xx2r\" (UID: \"9cd92df4-350d-4fe2-bd41-48b57198a268\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" Nov 03 00:23:19.014047 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:19.013674 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cd92df4-350d-4fe2-bd41-48b57198a268-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5xx2r\" (UID: \"9cd92df4-350d-4fe2-bd41-48b57198a268\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" Nov 03 00:23:19.014047 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:19.013716 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd92df4-350d-4fe2-bd41-48b57198a268-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5xx2r\" (UID: \"9cd92df4-350d-4fe2-bd41-48b57198a268\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" Nov 03 00:23:19.014047 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:19.013772 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cd92df4-350d-4fe2-bd41-48b57198a268-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5xx2r\" (UID: \"9cd92df4-350d-4fe2-bd41-48b57198a268\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" Nov 03 00:23:19.014047 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:19.013856 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7bql\" (UniqueName: \"kubernetes.io/projected/9cd92df4-350d-4fe2-bd41-48b57198a268-kube-api-access-f7bql\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5xx2r\" (UID: \"9cd92df4-350d-4fe2-bd41-48b57198a268\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" Nov 03 00:23:19.014047 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:19.013923 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cd92df4-350d-4fe2-bd41-48b57198a268-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5xx2r\" (UID: \"9cd92df4-350d-4fe2-bd41-48b57198a268\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" Nov 03 00:23:19.069856 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:19.069822 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r"] Nov 03 00:23:19.116498 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:19.116461 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cd92df4-350d-4fe2-bd41-48b57198a268-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5xx2r\" (UID: \"9cd92df4-350d-4fe2-bd41-48b57198a268\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" Nov 03 00:23:19.117186 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:19.117158 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd92df4-350d-4fe2-bd41-48b57198a268-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5xx2r\" (UID: \"9cd92df4-350d-4fe2-bd41-48b57198a268\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" Nov 03 00:23:19.117806 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:19.117787 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cd92df4-350d-4fe2-bd41-48b57198a268-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5xx2r\" (UID: \"9cd92df4-350d-4fe2-bd41-48b57198a268\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" Nov 03 00:23:19.118303 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:19.118283 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd92df4-350d-4fe2-bd41-48b57198a268-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5xx2r\" (UID: \"9cd92df4-350d-4fe2-bd41-48b57198a268\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" Nov 03 00:23:19.118761 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:19.118745 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cd92df4-350d-4fe2-bd41-48b57198a268-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5xx2r\" (UID: \"9cd92df4-350d-4fe2-bd41-48b57198a268\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" Nov 03 00:23:19.119409 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:19.119388 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7bql\" (UniqueName: \"kubernetes.io/projected/9cd92df4-350d-4fe2-bd41-48b57198a268-kube-api-access-f7bql\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5xx2r\" (UID: \"9cd92df4-350d-4fe2-bd41-48b57198a268\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" Nov 03 00:23:19.119872 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:19.119851 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cd92df4-350d-4fe2-bd41-48b57198a268-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5xx2r\" (UID: \"9cd92df4-350d-4fe2-bd41-48b57198a268\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" Nov 03 00:23:19.122418 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:19.122394 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cd92df4-350d-4fe2-bd41-48b57198a268-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5xx2r\" (UID: \"9cd92df4-350d-4fe2-bd41-48b57198a268\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" Nov 03 00:23:19.122583 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:19.122563 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cd92df4-350d-4fe2-bd41-48b57198a268-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5xx2r\" (UID: \"9cd92df4-350d-4fe2-bd41-48b57198a268\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" Nov 03 00:23:19.122783 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:19.120678 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cd92df4-350d-4fe2-bd41-48b57198a268-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5xx2r\" (UID: \"9cd92df4-350d-4fe2-bd41-48b57198a268\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" Nov 03 00:23:19.122950 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:19.118218 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cd92df4-350d-4fe2-bd41-48b57198a268-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5xx2r\" (UID: \"9cd92df4-350d-4fe2-bd41-48b57198a268\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" Nov 03 00:23:19.123359 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:19.123337 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cd92df4-350d-4fe2-bd41-48b57198a268-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5xx2r\" (UID: \"9cd92df4-350d-4fe2-bd41-48b57198a268\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" Nov 03 00:23:19.123487 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:19.117730 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd92df4-350d-4fe2-bd41-48b57198a268-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5xx2r\" (UID: \"9cd92df4-350d-4fe2-bd41-48b57198a268\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" Nov 03 00:23:19.126955 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:19.118670 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd92df4-350d-4fe2-bd41-48b57198a268-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5xx2r\" (UID: \"9cd92df4-350d-4fe2-bd41-48b57198a268\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" Nov 03 00:23:19.130212 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:19.130189 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cd92df4-350d-4fe2-bd41-48b57198a268-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5xx2r\" (UID: \"9cd92df4-350d-4fe2-bd41-48b57198a268\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" Nov 03 00:23:19.138750 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:19.138722 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cd92df4-350d-4fe2-bd41-48b57198a268-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5xx2r\" (UID: \"9cd92df4-350d-4fe2-bd41-48b57198a268\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" Nov 03 00:23:19.174771 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:19.174703 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cd92df4-350d-4fe2-bd41-48b57198a268-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5xx2r\" (UID: \"9cd92df4-350d-4fe2-bd41-48b57198a268\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" Nov 03 00:23:19.178513 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:19.178450 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7bql\" (UniqueName: \"kubernetes.io/projected/9cd92df4-350d-4fe2-bd41-48b57198a268-kube-api-access-f7bql\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5xx2r\" (UID: \"9cd92df4-350d-4fe2-bd41-48b57198a268\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" Nov 03 00:23:19.253304 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:19.253212 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" Nov 03 00:23:19.751055 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:19.743470 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:23:19.751055 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:19.743638 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:23:19.751055 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:19.743737 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:23:19.758186 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:19.758157 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r"] Nov 03 00:23:20.599807 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:20.599768 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" event={"ID":"9cd92df4-350d-4fe2-bd41-48b57198a268","Type":"ContainerStarted","Data":"cb15b08dca703176dd477987c8392821ee14f78e543279c0bfa955bee8df2fc5"} Nov 03 00:23:20.600249 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:20.599823 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" event={"ID":"9cd92df4-350d-4fe2-bd41-48b57198a268","Type":"ContainerStarted","Data":"44137fe013fe05046ec44e5a42b291681118412f63c6bb6ddae74da0a17d19fe"} Nov 03 00:23:20.784390 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:20.784331 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" podStartSLOduration=2.784318366 podStartE2EDuration="2.784318366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:23:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:23:20.759048008 +0000 UTC m=+5449.727579820" watchObservedRunningTime="2026-11-03 00:23:20.784318366 +0000 UTC m=+5449.752850215" Nov 03 00:23:21.253533 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:21.253501 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" Nov 03 00:23:21.255178 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:21.255149 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 03 00:23:21.255333 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:21.255197 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" podUID="9cd92df4-350d-4fe2-bd41-48b57198a268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 03 00:23:22.253708 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:22.253667 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 03 00:23:22.254142 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:22.253734 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" podUID="9cd92df4-350d-4fe2-bd41-48b57198a268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 03 00:23:23.254173 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:23.254136 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 03 00:23:23.254594 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:23.254192 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" podUID="9cd92df4-350d-4fe2-bd41-48b57198a268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 03 00:23:24.253540 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:24.253499 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 03 00:23:24.253753 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:24.253552 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" podUID="9cd92df4-350d-4fe2-bd41-48b57198a268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 03 00:23:25.176129 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:25.176089 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wx6l5"] Nov 03 00:23:25.185146 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:25.185116 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wx6l5" Nov 03 00:23:25.234951 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:25.234729 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wx6l5"] Nov 03 00:23:25.253727 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:25.253684 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 03 00:23:25.253914 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:25.253753 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" podUID="9cd92df4-350d-4fe2-bd41-48b57198a268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 03 00:23:25.278667 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:25.278635 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:23:25.278854 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:25.278820 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:23:25.289153 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:25.289129 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0904c42b-d66f-4f78-bf45-51dd67bf969b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-wx6l5\" (UID: \"0904c42b-d66f-4f78-bf45-51dd67bf969b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wx6l5" Nov 03 00:23:25.289257 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:25.289196 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxvrj\" (UniqueName: \"kubernetes.io/projected/0904c42b-d66f-4f78-bf45-51dd67bf969b-kube-api-access-pxvrj\") pod \"maas-api-revoke-sub-f03629db-wx6l5\" (UID: \"0904c42b-d66f-4f78-bf45-51dd67bf969b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wx6l5" Nov 03 00:23:25.289373 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:25.289350 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0904c42b-d66f-4f78-bf45-51dd67bf969b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-wx6l5\" (UID: \"0904c42b-d66f-4f78-bf45-51dd67bf969b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wx6l5" Nov 03 00:23:25.389883 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:25.389848 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxvrj\" (UniqueName: \"kubernetes.io/projected/0904c42b-d66f-4f78-bf45-51dd67bf969b-kube-api-access-pxvrj\") pod \"maas-api-revoke-sub-f03629db-wx6l5\" (UID: \"0904c42b-d66f-4f78-bf45-51dd67bf969b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wx6l5" Nov 03 00:23:25.390105 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:25.389957 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0904c42b-d66f-4f78-bf45-51dd67bf969b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-wx6l5\" (UID: \"0904c42b-d66f-4f78-bf45-51dd67bf969b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wx6l5" Nov 03 00:23:25.390105 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:25.390004 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0904c42b-d66f-4f78-bf45-51dd67bf969b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-wx6l5\" (UID: \"0904c42b-d66f-4f78-bf45-51dd67bf969b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wx6l5" Nov 03 00:23:25.390733 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:25.390711 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0904c42b-d66f-4f78-bf45-51dd67bf969b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-wx6l5\" (UID: \"0904c42b-d66f-4f78-bf45-51dd67bf969b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wx6l5" Nov 03 00:23:25.392328 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:25.392306 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0904c42b-d66f-4f78-bf45-51dd67bf969b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-wx6l5\" (UID: \"0904c42b-d66f-4f78-bf45-51dd67bf969b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wx6l5" Nov 03 00:23:25.445120 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:25.445086 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxvrj\" (UniqueName: \"kubernetes.io/projected/0904c42b-d66f-4f78-bf45-51dd67bf969b-kube-api-access-pxvrj\") pod \"maas-api-revoke-sub-f03629db-wx6l5\" (UID: \"0904c42b-d66f-4f78-bf45-51dd67bf969b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wx6l5" Nov 03 00:23:25.499018 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:25.498986 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wx6l5" Nov 03 00:23:25.902912 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:23:25.902868 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0904c42b_d66f_4f78_bf45_51dd67bf969b.slice/crio-97216a273cb104e4dcd9ae4585f58d0604e73049bf51bc03ef900324837962bb WatchSource:0}: Error finding container 97216a273cb104e4dcd9ae4585f58d0604e73049bf51bc03ef900324837962bb: Status 404 returned error can't find the container with id 97216a273cb104e4dcd9ae4585f58d0604e73049bf51bc03ef900324837962bb Nov 03 00:23:25.903282 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:25.903251 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wx6l5"] Nov 03 00:23:26.253969 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:26.253943 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 03 00:23:26.254325 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:26.253987 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" podUID="9cd92df4-350d-4fe2-bd41-48b57198a268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 03 00:23:26.654608 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:26.654512 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wx6l5" event={"ID":"0904c42b-d66f-4f78-bf45-51dd67bf969b","Type":"ContainerStarted","Data":"7b23d5b7e6d69d4c86d65f1ea68cc2c2f2e048d24efab6ca9a71adcc063e385d"} Nov 03 00:23:26.654608 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:26.654575 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wx6l5" event={"ID":"0904c42b-d66f-4f78-bf45-51dd67bf969b","Type":"ContainerStarted","Data":"97216a273cb104e4dcd9ae4585f58d0604e73049bf51bc03ef900324837962bb"} Nov 03 00:23:26.770853 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:26.770796 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wx6l5" podStartSLOduration=1.770782419 podStartE2EDuration="1.770782419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:23:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:23:26.768634224 +0000 UTC m=+5455.737166012" watchObservedRunningTime="2026-11-03 00:23:26.770782419 +0000 UTC m=+5455.739314194" Nov 03 00:23:27.253586 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:27.253548 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 03 00:23:27.253758 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:27.253605 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" podUID="9cd92df4-350d-4fe2-bd41-48b57198a268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 03 00:23:27.664071 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:27.663978 5076 generic.go:356] "Generic (PLEG): container finished" podID="0904c42b-d66f-4f78-bf45-51dd67bf969b" containerID="7b23d5b7e6d69d4c86d65f1ea68cc2c2f2e048d24efab6ca9a71adcc063e385d" exitCode=0 Nov 03 00:23:27.664444 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:27.664063 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wx6l5" event={"ID":"0904c42b-d66f-4f78-bf45-51dd67bf969b","Type":"ContainerDied","Data":"7b23d5b7e6d69d4c86d65f1ea68cc2c2f2e048d24efab6ca9a71adcc063e385d"} Nov 03 00:23:28.272076 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:28.272038 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" Nov 03 00:23:28.272580 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:28.272547 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" Nov 03 00:23:28.273401 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:28.273373 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5xx2r" Nov 03 00:23:28.814932 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:28.814905 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wx6l5" Nov 03 00:23:28.926587 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:28.926486 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0904c42b-d66f-4f78-bf45-51dd67bf969b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0904c42b-d66f-4f78-bf45-51dd67bf969b-maas-api-openshift-ca\") pod \"0904c42b-d66f-4f78-bf45-51dd67bf969b\" (UID: \"0904c42b-d66f-4f78-bf45-51dd67bf969b\") " Nov 03 00:23:28.926587 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:28.926533 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0904c42b-d66f-4f78-bf45-51dd67bf969b-kube-api-access-pxvrj\" (UniqueName: \"kubernetes.io/projected/0904c42b-d66f-4f78-bf45-51dd67bf969b-kube-api-access-pxvrj\") pod \"0904c42b-d66f-4f78-bf45-51dd67bf969b\" (UID: \"0904c42b-d66f-4f78-bf45-51dd67bf969b\") " Nov 03 00:23:28.927013 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:28.926982 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0904c42b-d66f-4f78-bf45-51dd67bf969b-maas-api-openshift-ca" pod "0904c42b-d66f-4f78-bf45-51dd67bf969b" (UID: "0904c42b-d66f-4f78-bf45-51dd67bf969b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:23:28.927161 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:28.927143 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0904c42b-d66f-4f78-bf45-51dd67bf969b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0904c42b-d66f-4f78-bf45-51dd67bf969b-maas-api-xks-ca\") pod \"0904c42b-d66f-4f78-bf45-51dd67bf969b\" (UID: \"0904c42b-d66f-4f78-bf45-51dd67bf969b\") " Nov 03 00:23:28.927594 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:28.927573 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0904c42b-d66f-4f78-bf45-51dd67bf969b-maas-api-openshift-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:23:28.929473 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:28.929443 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0904c42b-d66f-4f78-bf45-51dd67bf969b-kube-api-access-pxvrj" pod "0904c42b-d66f-4f78-bf45-51dd67bf969b" (UID: "0904c42b-d66f-4f78-bf45-51dd67bf969b"). InnerVolumeSpecName "kube-api-access-pxvrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:23:28.929597 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:28.929573 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0904c42b-d66f-4f78-bf45-51dd67bf969b-maas-api-xks-ca" pod "0904c42b-d66f-4f78-bf45-51dd67bf969b" (UID: "0904c42b-d66f-4f78-bf45-51dd67bf969b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:23:29.028510 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:29.028473 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxvrj\" (UniqueName: \"kubernetes.io/projected/0904c42b-d66f-4f78-bf45-51dd67bf969b-kube-api-access-pxvrj\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:23:29.028510 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:29.028503 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0904c42b-d66f-4f78-bf45-51dd67bf969b-maas-api-xks-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:23:29.676213 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:29.676181 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wx6l5" Nov 03 00:23:29.676391 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:29.676207 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wx6l5" event={"ID":"0904c42b-d66f-4f78-bf45-51dd67bf969b","Type":"ContainerDied","Data":"97216a273cb104e4dcd9ae4585f58d0604e73049bf51bc03ef900324837962bb"} Nov 03 00:23:29.676391 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:29.676270 5076 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97216a273cb104e4dcd9ae4585f58d0604e73049bf51bc03ef900324837962bb" Nov 03 00:23:33.888638 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:33.888594 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t"] Nov 03 00:23:33.889270 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:33.889249 5076 cpu_manager.go:414] "containerMap: removing container" podUID="0904c42b-d66f-4f78-bf45-51dd67bf969b" containerName="revoke-keys" Nov 03 00:23:33.889366 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:33.889273 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0904c42b-d66f-4f78-bf45-51dd67bf969b" containerName="revoke-keys" Nov 03 00:23:33.889433 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:33.889394 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="0904c42b-d66f-4f78-bf45-51dd67bf969b" containerName="revoke-keys" Nov 03 00:23:33.893047 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:33.893022 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" Nov 03 00:23:33.978297 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:33.978256 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-slv2t\" (UID: \"4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" Nov 03 00:23:33.978297 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:33.978314 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-slv2t\" (UID: \"4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" Nov 03 00:23:33.979543 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:33.978408 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x76rr\" (UniqueName: \"kubernetes.io/projected/4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4-kube-api-access-x76rr\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-slv2t\" (UID: \"4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" Nov 03 00:23:33.979543 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:33.978466 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-slv2t\" (UID: \"4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" Nov 03 00:23:33.979543 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:33.978539 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-slv2t\" (UID: \"4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" Nov 03 00:23:33.979543 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:33.978610 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-slv2t\" (UID: \"4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" Nov 03 00:23:33.979543 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:33.978666 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-slv2t\" (UID: \"4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" Nov 03 00:23:33.979543 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:33.978689 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-slv2t\" (UID: \"4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" Nov 03 00:23:33.979543 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:33.978944 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-slv2t\" (UID: \"4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" Nov 03 00:23:33.983679 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:33.983647 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t"] Nov 03 00:23:34.080316 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:34.080277 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-slv2t\" (UID: \"4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" Nov 03 00:23:34.080548 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:34.080337 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-slv2t\" (UID: \"4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" Nov 03 00:23:34.080548 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:34.080358 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-slv2t\" (UID: \"4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" Nov 03 00:23:34.080548 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:34.080376 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x76rr\" (UniqueName: \"kubernetes.io/projected/4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4-kube-api-access-x76rr\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-slv2t\" (UID: \"4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" Nov 03 00:23:34.080548 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:34.080401 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-slv2t\" (UID: \"4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" Nov 03 00:23:34.080548 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:34.080425 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-slv2t\" (UID: \"4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" Nov 03 00:23:34.080548 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:34.080481 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-slv2t\" (UID: \"4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" Nov 03 00:23:34.080548 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:34.080532 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-slv2t\" (UID: \"4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" Nov 03 00:23:34.080958 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:34.080553 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-slv2t\" (UID: \"4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" Nov 03 00:23:34.080958 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:34.080745 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-slv2t\" (UID: \"4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" Nov 03 00:23:34.080958 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:34.080835 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-slv2t\" (UID: \"4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" Nov 03 00:23:34.081086 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:34.081042 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-slv2t\" (UID: \"4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" Nov 03 00:23:34.081130 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:34.081113 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-slv2t\" (UID: \"4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" Nov 03 00:23:34.081234 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:34.081215 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-slv2t\" (UID: \"4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" Nov 03 00:23:34.082879 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:34.082852 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-slv2t\" (UID: \"4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" Nov 03 00:23:34.083108 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:34.083089 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-slv2t\" (UID: \"4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" Nov 03 00:23:34.136006 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:34.135967 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-slv2t\" (UID: \"4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" Nov 03 00:23:34.136192 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:34.136165 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x76rr\" (UniqueName: \"kubernetes.io/projected/4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4-kube-api-access-x76rr\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-slv2t\" (UID: \"4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" Nov 03 00:23:34.215866 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:34.215828 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" Nov 03 00:23:34.646936 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:34.646903 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t"] Nov 03 00:23:34.647795 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:23:34.647766 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b14bc8b_9e90_4e86_b679_a16fbf1f9ba4.slice/crio-569729939af055e3a73eed38d6248ff525bc2ac518ae956353a1f3ba25e7490f WatchSource:0}: Error finding container 569729939af055e3a73eed38d6248ff525bc2ac518ae956353a1f3ba25e7490f: Status 404 returned error can't find the container with id 569729939af055e3a73eed38d6248ff525bc2ac518ae956353a1f3ba25e7490f Nov 03 00:23:34.650716 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:34.650684 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:23:34.650817 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:34.650756 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:23:34.650817 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:34.650803 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:23:34.704651 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:34.704622 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" event={"ID":"4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4","Type":"ContainerStarted","Data":"569729939af055e3a73eed38d6248ff525bc2ac518ae956353a1f3ba25e7490f"} Nov 03 00:23:35.729915 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:35.723196 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" event={"ID":"4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4","Type":"ContainerStarted","Data":"a4e9dab1a38a247a91330d3567115672e913b997a0719b01773a4eaf41965cb0"} Nov 03 00:23:35.940837 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:35.940767 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" podStartSLOduration=2.940750959 podStartE2EDuration="2.940750959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:23:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:23:35.938753548 +0000 UTC m=+5464.907285332" watchObservedRunningTime="2026-11-03 00:23:35.940750959 +0000 UTC m=+5464.909282742" Nov 03 00:23:36.229119 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:36.229063 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" Nov 03 00:23:36.233914 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:36.233157 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-slv2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Nov 03 00:23:36.233914 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:36.233210 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" podUID="4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Nov 03 00:23:37.225925 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:37.225268 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-slv2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Nov 03 00:23:37.225925 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:37.225329 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" podUID="4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Nov 03 00:23:38.229915 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:38.217258 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-slv2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Nov 03 00:23:38.229915 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:38.217322 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" podUID="4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Nov 03 00:23:39.229915 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:39.229333 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-slv2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Nov 03 00:23:39.229915 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:39.229410 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" podUID="4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Nov 03 00:23:39.883019 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:23:39.882983 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:23:40.228916 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:40.220316 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-slv2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Nov 03 00:23:40.228916 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:40.220392 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" podUID="4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Nov 03 00:23:41.216967 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:41.216883 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-slv2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Nov 03 00:23:41.217478 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:41.216996 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" podUID="4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Nov 03 00:23:42.216863 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:42.216820 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-slv2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Nov 03 00:23:42.217057 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:42.216920 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" podUID="4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Nov 03 00:23:44.225915 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:44.221991 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-slv2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:23:44.225915 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:44.222056 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" podUID="4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:23:45.276915 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:45.273011 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-slv2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:23:45.276915 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:45.273130 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" podUID="4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:23:46.195358 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:46.195326 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" Nov 03 00:23:46.196345 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:46.196316 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" Nov 03 00:23:46.197078 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:46.197051 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-slv2t" Nov 03 00:23:46.223667 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:46.223637 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-84j9c"] Nov 03 00:23:46.227584 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:46.227562 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-84j9c" Nov 03 00:23:46.279277 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:46.279244 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-84j9c"] Nov 03 00:23:46.293026 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:46.292996 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:23:46.293235 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:46.293046 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:23:46.347527 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:46.347484 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69bdd70a-8148-4bac-acf1-4acd28b13f48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-84j9c\" (UID: \"69bdd70a-8148-4bac-acf1-4acd28b13f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-84j9c" Nov 03 00:23:46.347727 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:46.347549 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jxt4\" (UniqueName: \"kubernetes.io/projected/69bdd70a-8148-4bac-acf1-4acd28b13f48-kube-api-access-6jxt4\") pod \"maas-api-revoke-sub-0165d7d2-84j9c\" (UID: \"69bdd70a-8148-4bac-acf1-4acd28b13f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-84j9c" Nov 03 00:23:46.347806 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:46.347735 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69bdd70a-8148-4bac-acf1-4acd28b13f48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-84j9c\" (UID: \"69bdd70a-8148-4bac-acf1-4acd28b13f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-84j9c" Nov 03 00:23:46.449164 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:46.449062 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69bdd70a-8148-4bac-acf1-4acd28b13f48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-84j9c\" (UID: \"69bdd70a-8148-4bac-acf1-4acd28b13f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-84j9c" Nov 03 00:23:46.449164 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:46.449136 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jxt4\" (UniqueName: \"kubernetes.io/projected/69bdd70a-8148-4bac-acf1-4acd28b13f48-kube-api-access-6jxt4\") pod \"maas-api-revoke-sub-0165d7d2-84j9c\" (UID: \"69bdd70a-8148-4bac-acf1-4acd28b13f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-84j9c" Nov 03 00:23:46.449394 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:46.449206 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69bdd70a-8148-4bac-acf1-4acd28b13f48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-84j9c\" (UID: \"69bdd70a-8148-4bac-acf1-4acd28b13f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-84j9c" Nov 03 00:23:46.450549 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:46.450519 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69bdd70a-8148-4bac-acf1-4acd28b13f48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-84j9c\" (UID: \"69bdd70a-8148-4bac-acf1-4acd28b13f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-84j9c" Nov 03 00:23:46.453229 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:46.453206 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69bdd70a-8148-4bac-acf1-4acd28b13f48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-84j9c\" (UID: \"69bdd70a-8148-4bac-acf1-4acd28b13f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-84j9c" Nov 03 00:23:46.504354 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:46.504324 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jxt4\" (UniqueName: \"kubernetes.io/projected/69bdd70a-8148-4bac-acf1-4acd28b13f48-kube-api-access-6jxt4\") pod \"maas-api-revoke-sub-0165d7d2-84j9c\" (UID: \"69bdd70a-8148-4bac-acf1-4acd28b13f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-84j9c" Nov 03 00:23:46.542573 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:46.542542 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-84j9c" Nov 03 00:23:46.912086 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:46.912057 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-84j9c"] Nov 03 00:23:46.913534 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:23:46.913508 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69bdd70a_8148_4bac_acf1_4acd28b13f48.slice/crio-35f474b11af48e235c2888faec9d5756f1206ea27c841705f24c9b3563e4c721 WatchSource:0}: Error finding container 35f474b11af48e235c2888faec9d5756f1206ea27c841705f24c9b3563e4c721: Status 404 returned error can't find the container with id 35f474b11af48e235c2888faec9d5756f1206ea27c841705f24c9b3563e4c721 Nov 03 00:23:47.811571 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:47.811539 5076 generic.go:356] "Generic (PLEG): container finished" podID="69bdd70a-8148-4bac-acf1-4acd28b13f48" containerID="9a0621e72819c1a1123df364e0bca8560d9c812e94753359b75b980c652c8e66" exitCode=0 Nov 03 00:23:47.812027 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:47.811579 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-84j9c" event={"ID":"69bdd70a-8148-4bac-acf1-4acd28b13f48","Type":"ContainerDied","Data":"9a0621e72819c1a1123df364e0bca8560d9c812e94753359b75b980c652c8e66"} Nov 03 00:23:47.812027 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:47.811633 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-84j9c" event={"ID":"69bdd70a-8148-4bac-acf1-4acd28b13f48","Type":"ContainerStarted","Data":"35f474b11af48e235c2888faec9d5756f1206ea27c841705f24c9b3563e4c721"} Nov 03 00:23:48.960607 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:48.960579 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-84j9c" Nov 03 00:23:49.079178 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:49.079101 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69bdd70a-8148-4bac-acf1-4acd28b13f48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69bdd70a-8148-4bac-acf1-4acd28b13f48-maas-api-openshift-ca\") pod \"69bdd70a-8148-4bac-acf1-4acd28b13f48\" (UID: \"69bdd70a-8148-4bac-acf1-4acd28b13f48\") " Nov 03 00:23:49.079359 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:49.079239 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69bdd70a-8148-4bac-acf1-4acd28b13f48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69bdd70a-8148-4bac-acf1-4acd28b13f48-maas-api-xks-ca\") pod \"69bdd70a-8148-4bac-acf1-4acd28b13f48\" (UID: \"69bdd70a-8148-4bac-acf1-4acd28b13f48\") " Nov 03 00:23:49.079359 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:49.079311 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69bdd70a-8148-4bac-acf1-4acd28b13f48-kube-api-access-6jxt4\" (UniqueName: \"kubernetes.io/projected/69bdd70a-8148-4bac-acf1-4acd28b13f48-kube-api-access-6jxt4\") pod \"69bdd70a-8148-4bac-acf1-4acd28b13f48\" (UID: \"69bdd70a-8148-4bac-acf1-4acd28b13f48\") " Nov 03 00:23:49.079560 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:49.079533 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69bdd70a-8148-4bac-acf1-4acd28b13f48-maas-api-openshift-ca" pod "69bdd70a-8148-4bac-acf1-4acd28b13f48" (UID: "69bdd70a-8148-4bac-acf1-4acd28b13f48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:23:49.079771 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:49.079750 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69bdd70a-8148-4bac-acf1-4acd28b13f48-maas-api-openshift-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:23:49.081351 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:49.081323 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69bdd70a-8148-4bac-acf1-4acd28b13f48-maas-api-xks-ca" pod "69bdd70a-8148-4bac-acf1-4acd28b13f48" (UID: "69bdd70a-8148-4bac-acf1-4acd28b13f48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:23:49.081463 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:49.081424 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69bdd70a-8148-4bac-acf1-4acd28b13f48-kube-api-access-6jxt4" pod "69bdd70a-8148-4bac-acf1-4acd28b13f48" (UID: "69bdd70a-8148-4bac-acf1-4acd28b13f48"). InnerVolumeSpecName "kube-api-access-6jxt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:23:49.180969 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:49.180931 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69bdd70a-8148-4bac-acf1-4acd28b13f48-maas-api-xks-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:23:49.180969 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:49.180957 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jxt4\" (UniqueName: \"kubernetes.io/projected/69bdd70a-8148-4bac-acf1-4acd28b13f48-kube-api-access-6jxt4\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:23:49.822589 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:49.822555 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-84j9c" event={"ID":"69bdd70a-8148-4bac-acf1-4acd28b13f48","Type":"ContainerDied","Data":"35f474b11af48e235c2888faec9d5756f1206ea27c841705f24c9b3563e4c721"} Nov 03 00:23:49.822766 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:49.822609 5076 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35f474b11af48e235c2888faec9d5756f1206ea27c841705f24c9b3563e4c721" Nov 03 00:23:49.822766 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:49.822567 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-84j9c" Nov 03 00:23:58.075095 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:58.075056 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-czjqx"] Nov 03 00:23:58.084706 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:58.084668 5076 cpu_manager.go:414] "containerMap: removing container" podUID="69bdd70a-8148-4bac-acf1-4acd28b13f48" containerName="revoke-keys" Nov 03 00:23:58.084706 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:58.084704 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69bdd70a-8148-4bac-acf1-4acd28b13f48" containerName="revoke-keys" Nov 03 00:23:58.084922 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:58.084853 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="69bdd70a-8148-4bac-acf1-4acd28b13f48" containerName="revoke-keys" Nov 03 00:23:58.088709 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:58.088689 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-czjqx" Nov 03 00:23:58.131917 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:58.131863 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-czjqx"] Nov 03 00:23:58.170215 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:58.170174 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ab93ca-ad0b-468e-b906-3fae9b5b9be6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-czjqx\" (UID: \"65ab93ca-ad0b-468e-b906-3fae9b5b9be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-czjqx" Nov 03 00:23:58.170393 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:58.170234 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ab93ca-ad0b-468e-b906-3fae9b5b9be6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-czjqx\" (UID: \"65ab93ca-ad0b-468e-b906-3fae9b5b9be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-czjqx" Nov 03 00:23:58.170393 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:58.170288 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw8dd\" (UniqueName: \"kubernetes.io/projected/65ab93ca-ad0b-468e-b906-3fae9b5b9be6-kube-api-access-bw8dd\") pod \"maas-api-revoke-sub-6b19338f-czjqx\" (UID: \"65ab93ca-ad0b-468e-b906-3fae9b5b9be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-czjqx" Nov 03 00:23:58.176410 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:58.176379 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:23:58.176563 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:58.176496 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:23:58.271188 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:58.271148 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ab93ca-ad0b-468e-b906-3fae9b5b9be6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-czjqx\" (UID: \"65ab93ca-ad0b-468e-b906-3fae9b5b9be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-czjqx" Nov 03 00:23:58.271188 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:58.271191 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ab93ca-ad0b-468e-b906-3fae9b5b9be6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-czjqx\" (UID: \"65ab93ca-ad0b-468e-b906-3fae9b5b9be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-czjqx" Nov 03 00:23:58.271462 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:58.271209 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw8dd\" (UniqueName: \"kubernetes.io/projected/65ab93ca-ad0b-468e-b906-3fae9b5b9be6-kube-api-access-bw8dd\") pod \"maas-api-revoke-sub-6b19338f-czjqx\" (UID: \"65ab93ca-ad0b-468e-b906-3fae9b5b9be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-czjqx" Nov 03 00:23:58.271806 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:58.271785 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ab93ca-ad0b-468e-b906-3fae9b5b9be6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-czjqx\" (UID: \"65ab93ca-ad0b-468e-b906-3fae9b5b9be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-czjqx" Nov 03 00:23:58.273916 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:58.273827 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ab93ca-ad0b-468e-b906-3fae9b5b9be6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-czjqx\" (UID: \"65ab93ca-ad0b-468e-b906-3fae9b5b9be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-czjqx" Nov 03 00:23:58.326698 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:58.326625 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw8dd\" (UniqueName: \"kubernetes.io/projected/65ab93ca-ad0b-468e-b906-3fae9b5b9be6-kube-api-access-bw8dd\") pod \"maas-api-revoke-sub-6b19338f-czjqx\" (UID: \"65ab93ca-ad0b-468e-b906-3fae9b5b9be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-czjqx" Nov 03 00:23:58.402325 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:58.402291 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-czjqx" Nov 03 00:23:58.771390 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:58.771357 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-czjqx"] Nov 03 00:23:58.772826 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:23:58.772795 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65ab93ca_ad0b_468e_b906_3fae9b5b9be6.slice/crio-d00aef546a3ed21c713cc0aa56f809a8aec346fe050367fcb166879f4c3a121e WatchSource:0}: Error finding container d00aef546a3ed21c713cc0aa56f809a8aec346fe050367fcb166879f4c3a121e: Status 404 returned error can't find the container with id d00aef546a3ed21c713cc0aa56f809a8aec346fe050367fcb166879f4c3a121e Nov 03 00:23:58.868523 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:58.868487 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-czjqx" event={"ID":"65ab93ca-ad0b-468e-b906-3fae9b5b9be6","Type":"ContainerStarted","Data":"d00aef546a3ed21c713cc0aa56f809a8aec346fe050367fcb166879f4c3a121e"} Nov 03 00:23:59.875915 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:59.875793 5076 generic.go:356] "Generic (PLEG): container finished" podID="65ab93ca-ad0b-468e-b906-3fae9b5b9be6" containerID="7ff63cef444bcdb432517a537218f67999b8edeed2a8b807bf0b478b085f6605" exitCode=0 Nov 03 00:23:59.875915 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:23:59.875878 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-czjqx" event={"ID":"65ab93ca-ad0b-468e-b906-3fae9b5b9be6","Type":"ContainerDied","Data":"7ff63cef444bcdb432517a537218f67999b8edeed2a8b807bf0b478b085f6605"} Nov 03 00:24:01.057792 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:24:01.057760 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-czjqx" Nov 03 00:24:01.096086 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:24:01.096051 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65ab93ca-ad0b-468e-b906-3fae9b5b9be6-kube-api-access-bw8dd\" (UniqueName: \"kubernetes.io/projected/65ab93ca-ad0b-468e-b906-3fae9b5b9be6-kube-api-access-bw8dd\") pod \"65ab93ca-ad0b-468e-b906-3fae9b5b9be6\" (UID: \"65ab93ca-ad0b-468e-b906-3fae9b5b9be6\") " Nov 03 00:24:01.096243 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:24:01.096101 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65ab93ca-ad0b-468e-b906-3fae9b5b9be6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ab93ca-ad0b-468e-b906-3fae9b5b9be6-maas-api-openshift-ca\") pod \"65ab93ca-ad0b-468e-b906-3fae9b5b9be6\" (UID: \"65ab93ca-ad0b-468e-b906-3fae9b5b9be6\") " Nov 03 00:24:01.096243 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:24:01.096154 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65ab93ca-ad0b-468e-b906-3fae9b5b9be6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ab93ca-ad0b-468e-b906-3fae9b5b9be6-maas-api-xks-ca\") pod \"65ab93ca-ad0b-468e-b906-3fae9b5b9be6\" (UID: \"65ab93ca-ad0b-468e-b906-3fae9b5b9be6\") " Nov 03 00:24:01.096570 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:24:01.096544 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65ab93ca-ad0b-468e-b906-3fae9b5b9be6-maas-api-openshift-ca" pod "65ab93ca-ad0b-468e-b906-3fae9b5b9be6" (UID: "65ab93ca-ad0b-468e-b906-3fae9b5b9be6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:24:01.098178 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:24:01.098150 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65ab93ca-ad0b-468e-b906-3fae9b5b9be6-maas-api-xks-ca" pod "65ab93ca-ad0b-468e-b906-3fae9b5b9be6" (UID: "65ab93ca-ad0b-468e-b906-3fae9b5b9be6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:24:01.098324 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:24:01.098297 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65ab93ca-ad0b-468e-b906-3fae9b5b9be6-kube-api-access-bw8dd" pod "65ab93ca-ad0b-468e-b906-3fae9b5b9be6" (UID: "65ab93ca-ad0b-468e-b906-3fae9b5b9be6"). InnerVolumeSpecName "kube-api-access-bw8dd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:24:01.197717 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:24:01.197678 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw8dd\" (UniqueName: \"kubernetes.io/projected/65ab93ca-ad0b-468e-b906-3fae9b5b9be6-kube-api-access-bw8dd\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:24:01.197717 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:24:01.197708 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ab93ca-ad0b-468e-b906-3fae9b5b9be6-maas-api-openshift-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:24:01.197717 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:24:01.197720 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ab93ca-ad0b-468e-b906-3fae9b5b9be6-maas-api-xks-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:24:01.888412 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:24:01.888383 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-czjqx" Nov 03 00:24:01.888412 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:24:01.888390 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-czjqx" event={"ID":"65ab93ca-ad0b-468e-b906-3fae9b5b9be6","Type":"ContainerDied","Data":"d00aef546a3ed21c713cc0aa56f809a8aec346fe050367fcb166879f4c3a121e"} Nov 03 00:24:01.888637 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:24:01.888445 5076 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d00aef546a3ed21c713cc0aa56f809a8aec346fe050367fcb166879f4c3a121e" Nov 03 00:24:09.884065 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:24:09.884017 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:24:39.884006 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:24:39.883965 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:25:08.829481 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:08.829404 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2c9e7c5-w6wjx"] Nov 03 00:25:08.829914 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:08.829885 5076 cpu_manager.go:414] "containerMap: removing container" podUID="65ab93ca-ad0b-468e-b906-3fae9b5b9be6" containerName="revoke-keys" Nov 03 00:25:08.829966 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:08.829916 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65ab93ca-ad0b-468e-b906-3fae9b5b9be6" containerName="revoke-keys" Nov 03 00:25:08.830005 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:08.829997 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="65ab93ca-ad0b-468e-b906-3fae9b5b9be6" containerName="revoke-keys" Nov 03 00:25:08.833233 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:08.833217 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c9e7c5-w6wjx" Nov 03 00:25:08.886660 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:08.886622 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/185823d3-c448-42d6-ba42-fb7f86ce7cc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2c9e7c5-w6wjx\" (UID: \"185823d3-c448-42d6-ba42-fb7f86ce7cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c9e7c5-w6wjx" Nov 03 00:25:08.886831 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:08.886679 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/185823d3-c448-42d6-ba42-fb7f86ce7cc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2c9e7c5-w6wjx\" (UID: \"185823d3-c448-42d6-ba42-fb7f86ce7cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c9e7c5-w6wjx" Nov 03 00:25:08.886831 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:08.886741 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snhcb\" (UniqueName: \"kubernetes.io/projected/185823d3-c448-42d6-ba42-fb7f86ce7cc0-kube-api-access-snhcb\") pod \"maas-api-revoke-sub-e2c9e7c5-w6wjx\" (UID: \"185823d3-c448-42d6-ba42-fb7f86ce7cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c9e7c5-w6wjx" Nov 03 00:25:08.887624 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:08.887601 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2c9e7c5-w6wjx"] Nov 03 00:25:08.931313 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:08.931281 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:25:08.931482 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:08.931364 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:25:08.987456 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:08.987405 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/185823d3-c448-42d6-ba42-fb7f86ce7cc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2c9e7c5-w6wjx\" (UID: \"185823d3-c448-42d6-ba42-fb7f86ce7cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c9e7c5-w6wjx" Nov 03 00:25:08.987456 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:08.987445 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/185823d3-c448-42d6-ba42-fb7f86ce7cc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2c9e7c5-w6wjx\" (UID: \"185823d3-c448-42d6-ba42-fb7f86ce7cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c9e7c5-w6wjx" Nov 03 00:25:08.987728 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:08.987469 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snhcb\" (UniqueName: \"kubernetes.io/projected/185823d3-c448-42d6-ba42-fb7f86ce7cc0-kube-api-access-snhcb\") pod \"maas-api-revoke-sub-e2c9e7c5-w6wjx\" (UID: \"185823d3-c448-42d6-ba42-fb7f86ce7cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c9e7c5-w6wjx" Nov 03 00:25:08.988262 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:08.988237 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/185823d3-c448-42d6-ba42-fb7f86ce7cc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2c9e7c5-w6wjx\" (UID: \"185823d3-c448-42d6-ba42-fb7f86ce7cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c9e7c5-w6wjx" Nov 03 00:25:08.989861 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:08.989840 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/185823d3-c448-42d6-ba42-fb7f86ce7cc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2c9e7c5-w6wjx\" (UID: \"185823d3-c448-42d6-ba42-fb7f86ce7cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c9e7c5-w6wjx" Nov 03 00:25:09.044566 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:09.044535 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snhcb\" (UniqueName: \"kubernetes.io/projected/185823d3-c448-42d6-ba42-fb7f86ce7cc0-kube-api-access-snhcb\") pod \"maas-api-revoke-sub-e2c9e7c5-w6wjx\" (UID: \"185823d3-c448-42d6-ba42-fb7f86ce7cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c9e7c5-w6wjx" Nov 03 00:25:09.146254 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:09.146174 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c9e7c5-w6wjx" Nov 03 00:25:09.510679 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:09.510651 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2c9e7c5-w6wjx"] Nov 03 00:25:09.512577 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:25:09.512546 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod185823d3_c448_42d6_ba42_fb7f86ce7cc0.slice/crio-2a1ae50db92bac906544fc3de4e6a3857143f5e834af2f3909eda993f0339d53 WatchSource:0}: Error finding container 2a1ae50db92bac906544fc3de4e6a3857143f5e834af2f3909eda993f0339d53: Status 404 returned error can't find the container with id 2a1ae50db92bac906544fc3de4e6a3857143f5e834af2f3909eda993f0339d53 Nov 03 00:25:09.882665 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:25:09.882578 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:25:10.370113 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:10.370077 5076 generic.go:356] "Generic (PLEG): container finished" podID="185823d3-c448-42d6-ba42-fb7f86ce7cc0" containerID="bcee262a98f90f2d989294ef4a2337a6e8c7981b6cce065f732c86bbb937c3bf" exitCode=0 Nov 03 00:25:10.370312 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:10.370164 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c9e7c5-w6wjx" event={"ID":"185823d3-c448-42d6-ba42-fb7f86ce7cc0","Type":"ContainerDied","Data":"bcee262a98f90f2d989294ef4a2337a6e8c7981b6cce065f732c86bbb937c3bf"} Nov 03 00:25:10.370312 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:10.370229 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c9e7c5-w6wjx" event={"ID":"185823d3-c448-42d6-ba42-fb7f86ce7cc0","Type":"ContainerStarted","Data":"2a1ae50db92bac906544fc3de4e6a3857143f5e834af2f3909eda993f0339d53"} Nov 03 00:25:11.523068 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:11.523040 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c9e7c5-w6wjx" Nov 03 00:25:11.614026 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:11.613986 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/185823d3-c448-42d6-ba42-fb7f86ce7cc0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/185823d3-c448-42d6-ba42-fb7f86ce7cc0-maas-api-xks-ca\") pod \"185823d3-c448-42d6-ba42-fb7f86ce7cc0\" (UID: \"185823d3-c448-42d6-ba42-fb7f86ce7cc0\") " Nov 03 00:25:11.614230 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:11.614047 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/185823d3-c448-42d6-ba42-fb7f86ce7cc0-kube-api-access-snhcb\" (UniqueName: \"kubernetes.io/projected/185823d3-c448-42d6-ba42-fb7f86ce7cc0-kube-api-access-snhcb\") pod \"185823d3-c448-42d6-ba42-fb7f86ce7cc0\" (UID: \"185823d3-c448-42d6-ba42-fb7f86ce7cc0\") " Nov 03 00:25:11.614230 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:11.614123 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/185823d3-c448-42d6-ba42-fb7f86ce7cc0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/185823d3-c448-42d6-ba42-fb7f86ce7cc0-maas-api-openshift-ca\") pod \"185823d3-c448-42d6-ba42-fb7f86ce7cc0\" (UID: \"185823d3-c448-42d6-ba42-fb7f86ce7cc0\") " Nov 03 00:25:11.614514 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:11.614488 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/185823d3-c448-42d6-ba42-fb7f86ce7cc0-maas-api-openshift-ca" pod "185823d3-c448-42d6-ba42-fb7f86ce7cc0" (UID: "185823d3-c448-42d6-ba42-fb7f86ce7cc0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:25:11.616128 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:11.616107 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/185823d3-c448-42d6-ba42-fb7f86ce7cc0-maas-api-xks-ca" pod "185823d3-c448-42d6-ba42-fb7f86ce7cc0" (UID: "185823d3-c448-42d6-ba42-fb7f86ce7cc0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:25:11.616203 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:11.616183 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/185823d3-c448-42d6-ba42-fb7f86ce7cc0-kube-api-access-snhcb" pod "185823d3-c448-42d6-ba42-fb7f86ce7cc0" (UID: "185823d3-c448-42d6-ba42-fb7f86ce7cc0"). InnerVolumeSpecName "kube-api-access-snhcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:25:11.715663 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:11.715617 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/185823d3-c448-42d6-ba42-fb7f86ce7cc0-maas-api-xks-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:25:11.715663 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:11.715661 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snhcb\" (UniqueName: \"kubernetes.io/projected/185823d3-c448-42d6-ba42-fb7f86ce7cc0-kube-api-access-snhcb\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:25:11.715663 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:11.715672 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/185823d3-c448-42d6-ba42-fb7f86ce7cc0-maas-api-openshift-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:25:12.385140 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:12.385111 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c9e7c5-w6wjx" Nov 03 00:25:12.385326 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:12.385149 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c9e7c5-w6wjx" event={"ID":"185823d3-c448-42d6-ba42-fb7f86ce7cc0","Type":"ContainerDied","Data":"2a1ae50db92bac906544fc3de4e6a3857143f5e834af2f3909eda993f0339d53"} Nov 03 00:25:12.385326 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:12.385199 5076 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a1ae50db92bac906544fc3de4e6a3857143f5e834af2f3909eda993f0339d53" Nov 03 00:25:27.476305 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:27.476252 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fffe4bab-mcqm6"] Nov 03 00:25:27.477551 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:27.477510 5076 cpu_manager.go:414] "containerMap: removing container" podUID="185823d3-c448-42d6-ba42-fb7f86ce7cc0" containerName="revoke-keys" Nov 03 00:25:27.477732 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:27.477713 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="185823d3-c448-42d6-ba42-fb7f86ce7cc0" containerName="revoke-keys" Nov 03 00:25:27.478070 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:27.478051 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="185823d3-c448-42d6-ba42-fb7f86ce7cc0" containerName="revoke-keys" Nov 03 00:25:27.481920 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:27.481881 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffe4bab-mcqm6" Nov 03 00:25:27.534992 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:27.534945 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fffe4bab-mcqm6"] Nov 03 00:25:27.579956 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:27.577274 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:25:27.579956 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:27.577357 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:25:27.587061 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:27.586657 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/421ba8d5-6602-4aef-bcb5-9518f1843d33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fffe4bab-mcqm6\" (UID: \"421ba8d5-6602-4aef-bcb5-9518f1843d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffe4bab-mcqm6" Nov 03 00:25:27.587061 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:27.586752 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtfdw\" (UniqueName: \"kubernetes.io/projected/421ba8d5-6602-4aef-bcb5-9518f1843d33-kube-api-access-qtfdw\") pod \"maas-api-revoke-sub-fffe4bab-mcqm6\" (UID: \"421ba8d5-6602-4aef-bcb5-9518f1843d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffe4bab-mcqm6" Nov 03 00:25:27.587061 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:27.586823 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/421ba8d5-6602-4aef-bcb5-9518f1843d33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fffe4bab-mcqm6\" (UID: \"421ba8d5-6602-4aef-bcb5-9518f1843d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffe4bab-mcqm6" Nov 03 00:25:27.688023 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:27.687981 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/421ba8d5-6602-4aef-bcb5-9518f1843d33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fffe4bab-mcqm6\" (UID: \"421ba8d5-6602-4aef-bcb5-9518f1843d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffe4bab-mcqm6" Nov 03 00:25:27.688247 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:27.688079 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtfdw\" (UniqueName: \"kubernetes.io/projected/421ba8d5-6602-4aef-bcb5-9518f1843d33-kube-api-access-qtfdw\") pod \"maas-api-revoke-sub-fffe4bab-mcqm6\" (UID: \"421ba8d5-6602-4aef-bcb5-9518f1843d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffe4bab-mcqm6" Nov 03 00:25:27.688247 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:27.688149 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/421ba8d5-6602-4aef-bcb5-9518f1843d33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fffe4bab-mcqm6\" (UID: \"421ba8d5-6602-4aef-bcb5-9518f1843d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffe4bab-mcqm6" Nov 03 00:25:27.689026 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:27.688997 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/421ba8d5-6602-4aef-bcb5-9518f1843d33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fffe4bab-mcqm6\" (UID: \"421ba8d5-6602-4aef-bcb5-9518f1843d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffe4bab-mcqm6" Nov 03 00:25:27.691130 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:27.691106 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/421ba8d5-6602-4aef-bcb5-9518f1843d33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fffe4bab-mcqm6\" (UID: \"421ba8d5-6602-4aef-bcb5-9518f1843d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffe4bab-mcqm6" Nov 03 00:25:27.744014 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:27.743932 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtfdw\" (UniqueName: \"kubernetes.io/projected/421ba8d5-6602-4aef-bcb5-9518f1843d33-kube-api-access-qtfdw\") pod \"maas-api-revoke-sub-fffe4bab-mcqm6\" (UID: \"421ba8d5-6602-4aef-bcb5-9518f1843d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffe4bab-mcqm6" Nov 03 00:25:27.802228 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:27.801931 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffe4bab-mcqm6" Nov 03 00:25:28.206961 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:28.206473 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fffe4bab-mcqm6"] Nov 03 00:25:28.207601 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:25:28.207554 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod421ba8d5_6602_4aef_bcb5_9518f1843d33.slice/crio-f41411a36864a08df0a1bd61f356e684a43c3f01dc0971ba765ba2653f30bfce WatchSource:0}: Error finding container f41411a36864a08df0a1bd61f356e684a43c3f01dc0971ba765ba2653f30bfce: Status 404 returned error can't find the container with id f41411a36864a08df0a1bd61f356e684a43c3f01dc0971ba765ba2653f30bfce Nov 03 00:25:28.513384 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:28.513275 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffe4bab-mcqm6" event={"ID":"421ba8d5-6602-4aef-bcb5-9518f1843d33","Type":"ContainerStarted","Data":"f41411a36864a08df0a1bd61f356e684a43c3f01dc0971ba765ba2653f30bfce"} Nov 03 00:25:29.547922 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:29.547843 5076 generic.go:356] "Generic (PLEG): container finished" podID="421ba8d5-6602-4aef-bcb5-9518f1843d33" containerID="354a60448a79e26f3c78e7792d99c1e9139fce4b8353e644da52bb0a4c2af921" exitCode=0 Nov 03 00:25:29.548548 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:29.548499 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffe4bab-mcqm6" event={"ID":"421ba8d5-6602-4aef-bcb5-9518f1843d33","Type":"ContainerDied","Data":"354a60448a79e26f3c78e7792d99c1e9139fce4b8353e644da52bb0a4c2af921"} Nov 03 00:25:30.693392 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:30.693365 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffe4bab-mcqm6" Nov 03 00:25:30.708591 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:30.708565 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/421ba8d5-6602-4aef-bcb5-9518f1843d33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/421ba8d5-6602-4aef-bcb5-9518f1843d33-maas-api-xks-ca\") pod \"421ba8d5-6602-4aef-bcb5-9518f1843d33\" (UID: \"421ba8d5-6602-4aef-bcb5-9518f1843d33\") " Nov 03 00:25:30.708718 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:30.708656 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/421ba8d5-6602-4aef-bcb5-9518f1843d33-kube-api-access-qtfdw\" (UniqueName: \"kubernetes.io/projected/421ba8d5-6602-4aef-bcb5-9518f1843d33-kube-api-access-qtfdw\") pod \"421ba8d5-6602-4aef-bcb5-9518f1843d33\" (UID: \"421ba8d5-6602-4aef-bcb5-9518f1843d33\") " Nov 03 00:25:30.708718 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:30.708696 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/421ba8d5-6602-4aef-bcb5-9518f1843d33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/421ba8d5-6602-4aef-bcb5-9518f1843d33-maas-api-openshift-ca\") pod \"421ba8d5-6602-4aef-bcb5-9518f1843d33\" (UID: \"421ba8d5-6602-4aef-bcb5-9518f1843d33\") " Nov 03 00:25:30.709066 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:30.709033 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/421ba8d5-6602-4aef-bcb5-9518f1843d33-maas-api-openshift-ca" pod "421ba8d5-6602-4aef-bcb5-9518f1843d33" (UID: "421ba8d5-6602-4aef-bcb5-9518f1843d33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:25:30.710519 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:30.710496 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/421ba8d5-6602-4aef-bcb5-9518f1843d33-maas-api-xks-ca" pod "421ba8d5-6602-4aef-bcb5-9518f1843d33" (UID: "421ba8d5-6602-4aef-bcb5-9518f1843d33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:25:30.710635 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:30.710616 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/421ba8d5-6602-4aef-bcb5-9518f1843d33-kube-api-access-qtfdw" pod "421ba8d5-6602-4aef-bcb5-9518f1843d33" (UID: "421ba8d5-6602-4aef-bcb5-9518f1843d33"). InnerVolumeSpecName "kube-api-access-qtfdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:25:30.809870 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:30.809779 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/421ba8d5-6602-4aef-bcb5-9518f1843d33-maas-api-openshift-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:25:30.809870 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:30.809808 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/421ba8d5-6602-4aef-bcb5-9518f1843d33-maas-api-xks-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:25:30.809870 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:30.809835 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtfdw\" (UniqueName: \"kubernetes.io/projected/421ba8d5-6602-4aef-bcb5-9518f1843d33-kube-api-access-qtfdw\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:25:31.561740 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:31.561708 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffe4bab-mcqm6" Nov 03 00:25:31.561953 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:31.561735 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffe4bab-mcqm6" event={"ID":"421ba8d5-6602-4aef-bcb5-9518f1843d33","Type":"ContainerDied","Data":"f41411a36864a08df0a1bd61f356e684a43c3f01dc0971ba765ba2653f30bfce"} Nov 03 00:25:31.561953 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:31.561792 5076 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f41411a36864a08df0a1bd61f356e684a43c3f01dc0971ba765ba2653f30bfce" Nov 03 00:25:39.882956 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:25:39.882917 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:25:40.538505 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:40.538469 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr"] Nov 03 00:25:40.538933 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:40.538919 5076 cpu_manager.go:414] "containerMap: removing container" podUID="421ba8d5-6602-4aef-bcb5-9518f1843d33" containerName="revoke-keys" Nov 03 00:25:40.538987 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:40.538935 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="421ba8d5-6602-4aef-bcb5-9518f1843d33" containerName="revoke-keys" Nov 03 00:25:40.539050 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:40.539041 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="421ba8d5-6602-4aef-bcb5-9518f1843d33" containerName="revoke-keys" Nov 03 00:25:40.542372 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:40.542350 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" Nov 03 00:25:40.605712 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:40.605677 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2b862a2-53aa-4c9b-8571-102879f30f26-istio-envoy\") pod \"e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr\" (UID: \"a2b862a2-53aa-4c9b-8571-102879f30f26\") " pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" Nov 03 00:25:40.605712 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:40.605720 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2b862a2-53aa-4c9b-8571-102879f30f26-istio-data\") pod \"e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr\" (UID: \"a2b862a2-53aa-4c9b-8571-102879f30f26\") " pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" Nov 03 00:25:40.605997 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:40.605772 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2b862a2-53aa-4c9b-8571-102879f30f26-istio-podinfo\") pod \"e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr\" (UID: \"a2b862a2-53aa-4c9b-8571-102879f30f26\") " pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" Nov 03 00:25:40.605997 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:40.605832 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4c7s\" (UniqueName: \"kubernetes.io/projected/a2b862a2-53aa-4c9b-8571-102879f30f26-kube-api-access-n4c7s\") pod \"e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr\" (UID: \"a2b862a2-53aa-4c9b-8571-102879f30f26\") " pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" Nov 03 00:25:40.605997 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:40.605864 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2b862a2-53aa-4c9b-8571-102879f30f26-istiod-ca-cert\") pod \"e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr\" (UID: \"a2b862a2-53aa-4c9b-8571-102879f30f26\") " pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" Nov 03 00:25:40.605997 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:40.605885 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b862a2-53aa-4c9b-8571-102879f30f26-workload-socket\") pod \"e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr\" (UID: \"a2b862a2-53aa-4c9b-8571-102879f30f26\") " pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" Nov 03 00:25:40.605997 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:40.605949 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b862a2-53aa-4c9b-8571-102879f30f26-credential-socket\") pod \"e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr\" (UID: \"a2b862a2-53aa-4c9b-8571-102879f30f26\") " pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" Nov 03 00:25:40.606271 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:40.606016 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2b862a2-53aa-4c9b-8571-102879f30f26-istio-token\") pod \"e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr\" (UID: \"a2b862a2-53aa-4c9b-8571-102879f30f26\") " pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" Nov 03 00:25:40.606271 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:40.606039 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2b862a2-53aa-4c9b-8571-102879f30f26-workload-certs\") pod \"e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr\" (UID: \"a2b862a2-53aa-4c9b-8571-102879f30f26\") " pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" Nov 03 00:25:40.607655 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:40.607633 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-adc00e-openshift-default-dockercfg-krqw4\"" Nov 03 00:25:40.643136 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:40.643104 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr"] Nov 03 00:25:40.707198 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:40.707160 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2b862a2-53aa-4c9b-8571-102879f30f26-istio-envoy\") pod \"e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr\" (UID: \"a2b862a2-53aa-4c9b-8571-102879f30f26\") " pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" Nov 03 00:25:40.707405 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:40.707217 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2b862a2-53aa-4c9b-8571-102879f30f26-istio-data\") pod \"e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr\" (UID: \"a2b862a2-53aa-4c9b-8571-102879f30f26\") " pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" Nov 03 00:25:40.707405 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:40.707328 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2b862a2-53aa-4c9b-8571-102879f30f26-istio-podinfo\") pod \"e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr\" (UID: \"a2b862a2-53aa-4c9b-8571-102879f30f26\") " pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" Nov 03 00:25:40.707405 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:40.707370 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4c7s\" (UniqueName: \"kubernetes.io/projected/a2b862a2-53aa-4c9b-8571-102879f30f26-kube-api-access-n4c7s\") pod \"e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr\" (UID: \"a2b862a2-53aa-4c9b-8571-102879f30f26\") " pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" Nov 03 00:25:40.707600 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:40.707410 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2b862a2-53aa-4c9b-8571-102879f30f26-istiod-ca-cert\") pod \"e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr\" (UID: \"a2b862a2-53aa-4c9b-8571-102879f30f26\") " pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" Nov 03 00:25:40.707600 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:40.707446 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b862a2-53aa-4c9b-8571-102879f30f26-workload-socket\") pod \"e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr\" (UID: \"a2b862a2-53aa-4c9b-8571-102879f30f26\") " pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" Nov 03 00:25:40.707600 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:40.707470 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b862a2-53aa-4c9b-8571-102879f30f26-credential-socket\") pod \"e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr\" (UID: \"a2b862a2-53aa-4c9b-8571-102879f30f26\") " pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" Nov 03 00:25:40.707600 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:40.707506 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2b862a2-53aa-4c9b-8571-102879f30f26-istio-token\") pod \"e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr\" (UID: \"a2b862a2-53aa-4c9b-8571-102879f30f26\") " pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" Nov 03 00:25:40.707600 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:40.707533 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2b862a2-53aa-4c9b-8571-102879f30f26-workload-certs\") pod \"e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr\" (UID: \"a2b862a2-53aa-4c9b-8571-102879f30f26\") " pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" Nov 03 00:25:40.710537 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:40.710502 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2b862a2-53aa-4c9b-8571-102879f30f26-workload-certs\") pod \"e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr\" (UID: \"a2b862a2-53aa-4c9b-8571-102879f30f26\") " pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" Nov 03 00:25:40.713033 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:40.713006 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2b862a2-53aa-4c9b-8571-102879f30f26-istio-data\") pod \"e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr\" (UID: \"a2b862a2-53aa-4c9b-8571-102879f30f26\") " pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" Nov 03 00:25:40.713470 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:40.713449 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b862a2-53aa-4c9b-8571-102879f30f26-credential-socket\") pod \"e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr\" (UID: \"a2b862a2-53aa-4c9b-8571-102879f30f26\") " pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" Nov 03 00:25:40.713643 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:40.713593 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2b862a2-53aa-4c9b-8571-102879f30f26-istiod-ca-cert\") pod \"e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr\" (UID: \"a2b862a2-53aa-4c9b-8571-102879f30f26\") " pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" Nov 03 00:25:40.714100 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:40.714070 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b862a2-53aa-4c9b-8571-102879f30f26-workload-socket\") pod \"e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr\" (UID: \"a2b862a2-53aa-4c9b-8571-102879f30f26\") " pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" Nov 03 00:25:40.733211 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:40.733164 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2b862a2-53aa-4c9b-8571-102879f30f26-istio-podinfo\") pod \"e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr\" (UID: \"a2b862a2-53aa-4c9b-8571-102879f30f26\") " pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" Nov 03 00:25:40.734154 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:40.734125 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2b862a2-53aa-4c9b-8571-102879f30f26-istio-envoy\") pod \"e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr\" (UID: \"a2b862a2-53aa-4c9b-8571-102879f30f26\") " pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" Nov 03 00:25:40.768091 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:40.768055 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4c7s\" (UniqueName: \"kubernetes.io/projected/a2b862a2-53aa-4c9b-8571-102879f30f26-kube-api-access-n4c7s\") pod \"e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr\" (UID: \"a2b862a2-53aa-4c9b-8571-102879f30f26\") " pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" Nov 03 00:25:40.772818 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:40.772786 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2b862a2-53aa-4c9b-8571-102879f30f26-istio-token\") pod \"e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr\" (UID: \"a2b862a2-53aa-4c9b-8571-102879f30f26\") " pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" Nov 03 00:25:40.856082 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:40.855989 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" Nov 03 00:25:41.273077 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:41.273047 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr"] Nov 03 00:25:41.274075 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:25:41.274049 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2b862a2_53aa_4c9b_8571_102879f30f26.slice/crio-14cea9f8fc8671749d403f5e0d69a4cd6f476891e4f2126dcbea616b40c6c97d WatchSource:0}: Error finding container 14cea9f8fc8671749d403f5e0d69a4cd6f476891e4f2126dcbea616b40c6c97d: Status 404 returned error can't find the container with id 14cea9f8fc8671749d403f5e0d69a4cd6f476891e4f2126dcbea616b40c6c97d Nov 03 00:25:41.276306 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:41.276274 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:25:41.276391 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:41.276342 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:25:41.276391 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:41.276371 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:25:41.615145 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:41.615052 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" event={"ID":"a2b862a2-53aa-4c9b-8571-102879f30f26","Type":"ContainerStarted","Data":"486ee0860be529857341f75bcfbcd69d529966f2da1499c1004c49635cfc3839"} Nov 03 00:25:41.615145 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:41.615109 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" event={"ID":"a2b862a2-53aa-4c9b-8571-102879f30f26","Type":"ContainerStarted","Data":"14cea9f8fc8671749d403f5e0d69a4cd6f476891e4f2126dcbea616b40c6c97d"} Nov 03 00:25:41.732535 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:41.732480 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" podStartSLOduration=1.732464444 podStartE2EDuration="1.732464444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:41.729818028 +0000 UTC m=+5590.698349843" watchObservedRunningTime="2026-11-03 00:25:41.732464444 +0000 UTC m=+5590.700996227" Nov 03 00:25:41.856106 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:41.856072 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" Nov 03 00:25:42.861669 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:42.861640 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" Nov 03 00:25:43.655400 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:43.655228 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" Nov 03 00:25:43.664405 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:43.664127 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" Nov 03 00:25:51.696836 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:51.696800 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf"] Nov 03 00:25:51.700149 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:51.700125 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf" Nov 03 00:25:51.800789 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:51.800758 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 00:25:51.800789 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:51.800760 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-b7gfc\"" Nov 03 00:25:51.801072 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:51.800771 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-adc00e-serving-cert\"" Nov 03 00:25:51.803217 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:51.803195 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf"] Nov 03 00:25:51.864036 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:51.864003 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz9tc\" (UniqueName: \"kubernetes.io/projected/2787345f-875d-4c0d-bf19-0b3417aa5c01-kube-api-access-wz9tc\") pod \"maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf\" (UID: \"2787345f-875d-4c0d-bf19-0b3417aa5c01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf" Nov 03 00:25:51.864206 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:51.864066 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2787345f-875d-4c0d-bf19-0b3417aa5c01-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf\" (UID: \"2787345f-875d-4c0d-bf19-0b3417aa5c01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf" Nov 03 00:25:51.864206 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:51.864168 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2787345f-875d-4c0d-bf19-0b3417aa5c01-metrics-certs\") pod \"maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf\" (UID: \"2787345f-875d-4c0d-bf19-0b3417aa5c01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf" Nov 03 00:25:51.864317 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:51.864299 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2787345f-875d-4c0d-bf19-0b3417aa5c01-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf\" (UID: \"2787345f-875d-4c0d-bf19-0b3417aa5c01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf" Nov 03 00:25:51.965212 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:51.965124 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz9tc\" (UniqueName: \"kubernetes.io/projected/2787345f-875d-4c0d-bf19-0b3417aa5c01-kube-api-access-wz9tc\") pod \"maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf\" (UID: \"2787345f-875d-4c0d-bf19-0b3417aa5c01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf" Nov 03 00:25:51.965513 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:51.965475 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2787345f-875d-4c0d-bf19-0b3417aa5c01-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf\" (UID: \"2787345f-875d-4c0d-bf19-0b3417aa5c01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf" Nov 03 00:25:51.965645 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:51.965523 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2787345f-875d-4c0d-bf19-0b3417aa5c01-metrics-certs\") pod \"maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf\" (UID: \"2787345f-875d-4c0d-bf19-0b3417aa5c01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf" Nov 03 00:25:51.965645 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:25:51.965615 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-adc00e-serving-cert: secret "maas-api-e2e-worker-single-main-adc00e-serving-cert" not found Nov 03 00:25:51.965763 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:25:51.965700 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2787345f-875d-4c0d-bf19-0b3417aa5c01-maas-api-tls podName:2787345f-875d-4c0d-bf19-0b3417aa5c01 nodeName:}" failed. No retries permitted until 2026-11-03 00:25:52.465676729 +0000 UTC m=+5601.434208509 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2787345f-875d-4c0d-bf19-0b3417aa5c01-maas-api-tls") pod "maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf" (UID: "2787345f-875d-4c0d-bf19-0b3417aa5c01") : secret "maas-api-e2e-worker-single-main-adc00e-serving-cert" not found Nov 03 00:25:51.965933 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:51.965619 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2787345f-875d-4c0d-bf19-0b3417aa5c01-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf\" (UID: \"2787345f-875d-4c0d-bf19-0b3417aa5c01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf" Nov 03 00:25:51.966327 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:51.966309 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2787345f-875d-4c0d-bf19-0b3417aa5c01-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf\" (UID: \"2787345f-875d-4c0d-bf19-0b3417aa5c01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf" Nov 03 00:25:51.968347 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:51.968323 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2787345f-875d-4c0d-bf19-0b3417aa5c01-metrics-certs\") pod \"maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf\" (UID: \"2787345f-875d-4c0d-bf19-0b3417aa5c01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf" Nov 03 00:25:52.020484 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:52.020451 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz9tc\" (UniqueName: \"kubernetes.io/projected/2787345f-875d-4c0d-bf19-0b3417aa5c01-kube-api-access-wz9tc\") pod \"maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf\" (UID: \"2787345f-875d-4c0d-bf19-0b3417aa5c01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf" Nov 03 00:25:52.471544 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:52.471497 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2787345f-875d-4c0d-bf19-0b3417aa5c01-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf\" (UID: \"2787345f-875d-4c0d-bf19-0b3417aa5c01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf" Nov 03 00:25:52.473984 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:52.473960 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2787345f-875d-4c0d-bf19-0b3417aa5c01-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf\" (UID: \"2787345f-875d-4c0d-bf19-0b3417aa5c01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf" Nov 03 00:25:52.614541 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:52.614504 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf" Nov 03 00:25:52.982745 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:52.982713 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf"] Nov 03 00:25:52.984240 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:25:52.984212 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2787345f_875d_4c0d_bf19_0b3417aa5c01.slice/crio-213842da6c4cf9d1be492486bd711ddae38391e081ed3c841aeb9ae6323a4956 WatchSource:0}: Error finding container 213842da6c4cf9d1be492486bd711ddae38391e081ed3c841aeb9ae6323a4956: Status 404 returned error can't find the container with id 213842da6c4cf9d1be492486bd711ddae38391e081ed3c841aeb9ae6323a4956 Nov 03 00:25:53.714182 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:53.714137 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf" event={"ID":"2787345f-875d-4c0d-bf19-0b3417aa5c01","Type":"ContainerStarted","Data":"213842da6c4cf9d1be492486bd711ddae38391e081ed3c841aeb9ae6323a4956"} Nov 03 00:25:54.723104 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:54.723062 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf" event={"ID":"2787345f-875d-4c0d-bf19-0b3417aa5c01","Type":"ContainerStarted","Data":"d11c4dbad0241527986648e5869fc6944e4b3607437161063c3bc6d7e90cba60"} Nov 03 00:25:54.723550 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:25:54.723268 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf" Nov 03 00:26:00.737846 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:26:00.737814 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf" Nov 03 00:26:00.848494 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:26:00.848432 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf" podStartSLOduration=8.901267365 podStartE2EDuration="9.848412956s" totalImagesPullingTime="947.145591ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:54.83661094 +0000 UTC m=+5603.805142728" watchObservedRunningTime="2026-11-03 00:26:00.848412956 +0000 UTC m=+5609.816944741" Nov 03 00:26:09.806932 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:26:09.806864 5076 generic.go:356] "Generic (PLEG): container finished" podID="a2b862a2-53aa-4c9b-8571-102879f30f26" containerID="486ee0860be529857341f75bcfbcd69d529966f2da1499c1004c49635cfc3839" exitCode=0 Nov 03 00:26:09.807484 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:26:09.806938 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" event={"ID":"a2b862a2-53aa-4c9b-8571-102879f30f26","Type":"ContainerDied","Data":"486ee0860be529857341f75bcfbcd69d529966f2da1499c1004c49635cfc3839"} Nov 03 00:26:09.807484 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:26:09.807425 5076 scope.go:122] "RemoveContainer" containerID="486ee0860be529857341f75bcfbcd69d529966f2da1499c1004c49635cfc3839" Nov 03 00:26:09.808697 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:26:09.808672 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:26:09.808806 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:26:09.808750 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:26:09.808806 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:26:09.808796 5076 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:26:09.887220 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:26:09.887184 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:26:10.836456 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:26:10.836417 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" event={"ID":"a2b862a2-53aa-4c9b-8571-102879f30f26","Type":"ContainerStarted","Data":"d12c8a1eb27dcabea4246e73494c580e4ac950440607d282ea6008f70af7db6e"} Nov 03 00:26:10.856595 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:26:10.856563 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" Nov 03 00:26:10.857496 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:26:10.857471 5076 patch_prober.go:29] interesting pod/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 03 00:26:10.857619 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:26:10.857517 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" podUID="a2b862a2-53aa-4c9b-8571-102879f30f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 03 00:26:11.856540 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:26:11.856505 5076 patch_prober.go:29] interesting pod/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 03 00:26:11.856990 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:26:11.856555 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" podUID="a2b862a2-53aa-4c9b-8571-102879f30f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 03 00:26:12.860972 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:26:12.859506 5076 patch_prober.go:29] interesting pod/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 03 00:26:12.860972 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:26:12.859585 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" podUID="a2b862a2-53aa-4c9b-8571-102879f30f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 03 00:26:13.643157 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:26:13.643121 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" Nov 03 00:26:13.857049 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:26:13.857014 5076 patch_prober.go:29] interesting pod/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 03 00:26:13.857243 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:26:13.857073 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" podUID="a2b862a2-53aa-4c9b-8571-102879f30f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 03 00:26:14.857011 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:26:14.856977 5076 patch_prober.go:29] interesting pod/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 03 00:26:14.857499 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:26:14.857028 5076 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" podUID="a2b862a2-53aa-4c9b-8571-102879f30f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 03 00:26:15.926041 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:26:15.926010 5076 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" Nov 03 00:26:15.928411 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:26:15.928386 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" Nov 03 00:26:39.884179 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:26:39.884138 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:27:09.884978 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:27:09.884941 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:27:13.046787 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:13.046728 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-n5x8b"] Nov 03 00:27:13.092534 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:13.092500 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-n5x8b"] Nov 03 00:27:13.147803 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:13.147766 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d029e6d-414e-4328-ad80-635de1c9bb35" path="/var/lib/kubelet/pods/3d029e6d-414e-4328-ad80-635de1c9bb35/volumes" Nov 03 00:27:22.575424 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:22.575390 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf"] Nov 03 00:27:22.575868 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:22.575605 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf" podUID="2787345f-875d-4c0d-bf19-0b3417aa5c01" containerName="maas-api" containerID="cri-o://d11c4dbad0241527986648e5869fc6944e4b3607437161063c3bc6d7e90cba60" gracePeriod=30 Nov 03 00:27:23.846643 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:23.846615 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf" Nov 03 00:27:23.994650 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:23.994598 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2787345f-875d-4c0d-bf19-0b3417aa5c01-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2787345f-875d-4c0d-bf19-0b3417aa5c01-metrics-certs\") pod \"2787345f-875d-4c0d-bf19-0b3417aa5c01\" (UID: \"2787345f-875d-4c0d-bf19-0b3417aa5c01\") " Nov 03 00:27:23.994878 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:23.994713 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2787345f-875d-4c0d-bf19-0b3417aa5c01-kube-api-access-wz9tc\" (UniqueName: \"kubernetes.io/projected/2787345f-875d-4c0d-bf19-0b3417aa5c01-kube-api-access-wz9tc\") pod \"2787345f-875d-4c0d-bf19-0b3417aa5c01\" (UID: \"2787345f-875d-4c0d-bf19-0b3417aa5c01\") " Nov 03 00:27:23.994878 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:23.994752 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2787345f-875d-4c0d-bf19-0b3417aa5c01-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2787345f-875d-4c0d-bf19-0b3417aa5c01-maas-api-tls\") pod \"2787345f-875d-4c0d-bf19-0b3417aa5c01\" (UID: \"2787345f-875d-4c0d-bf19-0b3417aa5c01\") " Nov 03 00:27:23.995090 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:23.994905 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2787345f-875d-4c0d-bf19-0b3417aa5c01-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2787345f-875d-4c0d-bf19-0b3417aa5c01-openshift-service-ca\") pod \"2787345f-875d-4c0d-bf19-0b3417aa5c01\" (UID: \"2787345f-875d-4c0d-bf19-0b3417aa5c01\") " Nov 03 00:27:23.995255 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:23.995229 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2787345f-875d-4c0d-bf19-0b3417aa5c01-openshift-service-ca" pod "2787345f-875d-4c0d-bf19-0b3417aa5c01" (UID: "2787345f-875d-4c0d-bf19-0b3417aa5c01"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:27:23.995407 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:23.995386 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2787345f-875d-4c0d-bf19-0b3417aa5c01-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:27:23.996914 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:23.996858 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2787345f-875d-4c0d-bf19-0b3417aa5c01-metrics-certs" pod "2787345f-875d-4c0d-bf19-0b3417aa5c01" (UID: "2787345f-875d-4c0d-bf19-0b3417aa5c01"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:27:23.997004 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:23.996962 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2787345f-875d-4c0d-bf19-0b3417aa5c01-kube-api-access-wz9tc" pod "2787345f-875d-4c0d-bf19-0b3417aa5c01" (UID: "2787345f-875d-4c0d-bf19-0b3417aa5c01"). InnerVolumeSpecName "kube-api-access-wz9tc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:27:23.997004 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:23.996986 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2787345f-875d-4c0d-bf19-0b3417aa5c01-maas-api-tls" pod "2787345f-875d-4c0d-bf19-0b3417aa5c01" (UID: "2787345f-875d-4c0d-bf19-0b3417aa5c01"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:27:24.100315 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:24.096692 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz9tc\" (UniqueName: \"kubernetes.io/projected/2787345f-875d-4c0d-bf19-0b3417aa5c01-kube-api-access-wz9tc\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:27:24.100315 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:24.096744 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2787345f-875d-4c0d-bf19-0b3417aa5c01-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:27:24.100315 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:24.096761 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2787345f-875d-4c0d-bf19-0b3417aa5c01-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:27:24.254618 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:24.254521 5076 generic.go:356] "Generic (PLEG): container finished" podID="2787345f-875d-4c0d-bf19-0b3417aa5c01" containerID="d11c4dbad0241527986648e5869fc6944e4b3607437161063c3bc6d7e90cba60" exitCode=0 Nov 03 00:27:24.254618 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:24.254588 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf" Nov 03 00:27:24.254827 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:24.254606 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf" event={"ID":"2787345f-875d-4c0d-bf19-0b3417aa5c01","Type":"ContainerDied","Data":"d11c4dbad0241527986648e5869fc6944e4b3607437161063c3bc6d7e90cba60"} Nov 03 00:27:24.254827 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:24.254665 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf" event={"ID":"2787345f-875d-4c0d-bf19-0b3417aa5c01","Type":"ContainerDied","Data":"213842da6c4cf9d1be492486bd711ddae38391e081ed3c841aeb9ae6323a4956"} Nov 03 00:27:24.254827 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:24.254688 5076 scope.go:122] "RemoveContainer" containerID="d11c4dbad0241527986648e5869fc6944e4b3607437161063c3bc6d7e90cba60" Nov 03 00:27:24.264773 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:24.264749 5076 scope.go:122] "RemoveContainer" containerID="d11c4dbad0241527986648e5869fc6944e4b3607437161063c3bc6d7e90cba60" Nov 03 00:27:24.265082 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:27:24.265059 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d11c4dbad0241527986648e5869fc6944e4b3607437161063c3bc6d7e90cba60\": container with ID starting with d11c4dbad0241527986648e5869fc6944e4b3607437161063c3bc6d7e90cba60 not found: ID does not exist" containerID="d11c4dbad0241527986648e5869fc6944e4b3607437161063c3bc6d7e90cba60" Nov 03 00:27:24.265155 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:24.265089 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d11c4dbad0241527986648e5869fc6944e4b3607437161063c3bc6d7e90cba60"} err="failed to get container status \"d11c4dbad0241527986648e5869fc6944e4b3607437161063c3bc6d7e90cba60\": rpc error: code = NotFound desc = could not find container \"d11c4dbad0241527986648e5869fc6944e4b3607437161063c3bc6d7e90cba60\": container with ID starting with d11c4dbad0241527986648e5869fc6944e4b3607437161063c3bc6d7e90cba60 not found: ID does not exist" Nov 03 00:27:24.411617 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:24.411587 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf"] Nov 03 00:27:24.413829 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:24.413802 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-adc00e-67496cf5fd-t4vcf"] Nov 03 00:27:25.146696 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:25.146661 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2787345f-875d-4c0d-bf19-0b3417aa5c01" path="/var/lib/kubelet/pods/2787345f-875d-4c0d-bf19-0b3417aa5c01/volumes" Nov 03 00:27:32.255801 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:32.255773 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 03 00:27:32.354837 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:32.354810 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 03 00:27:39.884725 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:27:39.884680 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:27:54.022800 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:54.022693 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr"] Nov 03 00:27:54.023308 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:54.022936 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" podUID="a2b862a2-53aa-4c9b-8571-102879f30f26" containerName="istio-proxy" containerID="cri-o://d12c8a1eb27dcabea4246e73494c580e4ac950440607d282ea6008f70af7db6e" gracePeriod=30 Nov 03 00:27:57.043213 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:57.043175 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-6ml8x"] Nov 03 00:27:57.090468 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:57.090433 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-6ml8x"] Nov 03 00:27:57.147614 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:57.147572 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="568a402f-785b-4479-91c5-8ae0814e66c0" path="/var/lib/kubelet/pods/568a402f-785b-4479-91c5-8ae0814e66c0/volumes" Nov 03 00:27:59.279799 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.279772 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" Nov 03 00:27:59.358204 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.358120 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2b862a2-53aa-4c9b-8571-102879f30f26-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2b862a2-53aa-4c9b-8571-102879f30f26-istio-data\") pod \"a2b862a2-53aa-4c9b-8571-102879f30f26\" (UID: \"a2b862a2-53aa-4c9b-8571-102879f30f26\") " Nov 03 00:27:59.358204 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.358173 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2b862a2-53aa-4c9b-8571-102879f30f26-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b862a2-53aa-4c9b-8571-102879f30f26-workload-socket\") pod \"a2b862a2-53aa-4c9b-8571-102879f30f26\" (UID: \"a2b862a2-53aa-4c9b-8571-102879f30f26\") " Nov 03 00:27:59.358441 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.358216 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2b862a2-53aa-4c9b-8571-102879f30f26-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2b862a2-53aa-4c9b-8571-102879f30f26-workload-certs\") pod \"a2b862a2-53aa-4c9b-8571-102879f30f26\" (UID: \"a2b862a2-53aa-4c9b-8571-102879f30f26\") " Nov 03 00:27:59.358441 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.358242 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2b862a2-53aa-4c9b-8571-102879f30f26-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b862a2-53aa-4c9b-8571-102879f30f26-credential-socket\") pod \"a2b862a2-53aa-4c9b-8571-102879f30f26\" (UID: \"a2b862a2-53aa-4c9b-8571-102879f30f26\") " Nov 03 00:27:59.358441 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.358284 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2b862a2-53aa-4c9b-8571-102879f30f26-kube-api-access-n4c7s\" (UniqueName: \"kubernetes.io/projected/a2b862a2-53aa-4c9b-8571-102879f30f26-kube-api-access-n4c7s\") pod \"a2b862a2-53aa-4c9b-8571-102879f30f26\" (UID: \"a2b862a2-53aa-4c9b-8571-102879f30f26\") " Nov 03 00:27:59.358441 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.358340 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2b862a2-53aa-4c9b-8571-102879f30f26-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2b862a2-53aa-4c9b-8571-102879f30f26-istio-envoy\") pod \"a2b862a2-53aa-4c9b-8571-102879f30f26\" (UID: \"a2b862a2-53aa-4c9b-8571-102879f30f26\") " Nov 03 00:27:59.358441 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.358410 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2b862a2-53aa-4c9b-8571-102879f30f26-istio-data" pod "a2b862a2-53aa-4c9b-8571-102879f30f26" (UID: "a2b862a2-53aa-4c9b-8571-102879f30f26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:27:59.358737 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.358449 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2b862a2-53aa-4c9b-8571-102879f30f26-istio-token\" (UniqueName: \"kubernetes.io/projected/a2b862a2-53aa-4c9b-8571-102879f30f26-istio-token\") pod \"a2b862a2-53aa-4c9b-8571-102879f30f26\" (UID: \"a2b862a2-53aa-4c9b-8571-102879f30f26\") " Nov 03 00:27:59.358737 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.358503 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2b862a2-53aa-4c9b-8571-102879f30f26-workload-socket" pod "a2b862a2-53aa-4c9b-8571-102879f30f26" (UID: "a2b862a2-53aa-4c9b-8571-102879f30f26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:27:59.358737 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.358561 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2b862a2-53aa-4c9b-8571-102879f30f26-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2b862a2-53aa-4c9b-8571-102879f30f26-istiod-ca-cert\") pod \"a2b862a2-53aa-4c9b-8571-102879f30f26\" (UID: \"a2b862a2-53aa-4c9b-8571-102879f30f26\") " Nov 03 00:27:59.358737 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.358597 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2b862a2-53aa-4c9b-8571-102879f30f26-workload-certs" pod "a2b862a2-53aa-4c9b-8571-102879f30f26" (UID: "a2b862a2-53aa-4c9b-8571-102879f30f26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:27:59.358737 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.358614 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2b862a2-53aa-4c9b-8571-102879f30f26-credential-socket" pod "a2b862a2-53aa-4c9b-8571-102879f30f26" (UID: "a2b862a2-53aa-4c9b-8571-102879f30f26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:27:59.358737 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.358628 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2b862a2-53aa-4c9b-8571-102879f30f26-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2b862a2-53aa-4c9b-8571-102879f30f26-istio-podinfo\") pod \"a2b862a2-53aa-4c9b-8571-102879f30f26\" (UID: \"a2b862a2-53aa-4c9b-8571-102879f30f26\") " Nov 03 00:27:59.359109 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.358956 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2b862a2-53aa-4c9b-8571-102879f30f26-istiod-ca-cert" pod "a2b862a2-53aa-4c9b-8571-102879f30f26" (UID: "a2b862a2-53aa-4c9b-8571-102879f30f26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:27:59.359109 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.359018 5076 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2b862a2-53aa-4c9b-8571-102879f30f26-istio-data\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:27:59.359109 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.359039 5076 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b862a2-53aa-4c9b-8571-102879f30f26-workload-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:27:59.359109 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.359054 5076 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2b862a2-53aa-4c9b-8571-102879f30f26-workload-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:27:59.359109 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.359068 5076 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b862a2-53aa-4c9b-8571-102879f30f26-credential-socket\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:27:59.360711 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.360684 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2b862a2-53aa-4c9b-8571-102879f30f26-istio-envoy" pod "a2b862a2-53aa-4c9b-8571-102879f30f26" (UID: "a2b862a2-53aa-4c9b-8571-102879f30f26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:27:59.360862 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.360719 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2b862a2-53aa-4c9b-8571-102879f30f26-kube-api-access-n4c7s" pod "a2b862a2-53aa-4c9b-8571-102879f30f26" (UID: "a2b862a2-53aa-4c9b-8571-102879f30f26"). InnerVolumeSpecName "kube-api-access-n4c7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:27:59.361079 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.361055 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2b862a2-53aa-4c9b-8571-102879f30f26-istio-podinfo" pod "a2b862a2-53aa-4c9b-8571-102879f30f26" (UID: "a2b862a2-53aa-4c9b-8571-102879f30f26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:27:59.361152 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.361076 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2b862a2-53aa-4c9b-8571-102879f30f26-istio-token" pod "a2b862a2-53aa-4c9b-8571-102879f30f26" (UID: "a2b862a2-53aa-4c9b-8571-102879f30f26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:27:59.448955 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.448916 5076 generic.go:356] "Generic (PLEG): container finished" podID="a2b862a2-53aa-4c9b-8571-102879f30f26" containerID="d12c8a1eb27dcabea4246e73494c580e4ac950440607d282ea6008f70af7db6e" exitCode=0 Nov 03 00:27:59.449215 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.448974 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" Nov 03 00:27:59.449215 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.448986 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" event={"ID":"a2b862a2-53aa-4c9b-8571-102879f30f26","Type":"ContainerDied","Data":"d12c8a1eb27dcabea4246e73494c580e4ac950440607d282ea6008f70af7db6e"} Nov 03 00:27:59.449215 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.449048 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr" event={"ID":"a2b862a2-53aa-4c9b-8571-102879f30f26","Type":"ContainerDied","Data":"14cea9f8fc8671749d403f5e0d69a4cd6f476891e4f2126dcbea616b40c6c97d"} Nov 03 00:27:59.449215 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.449073 5076 scope.go:122] "RemoveContainer" containerID="d12c8a1eb27dcabea4246e73494c580e4ac950440607d282ea6008f70af7db6e" Nov 03 00:27:59.461677 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.461644 5076 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2b862a2-53aa-4c9b-8571-102879f30f26-istio-podinfo\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:27:59.461926 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.461868 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4c7s\" (UniqueName: \"kubernetes.io/projected/a2b862a2-53aa-4c9b-8571-102879f30f26-kube-api-access-n4c7s\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:27:59.461926 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.461922 5076 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2b862a2-53aa-4c9b-8571-102879f30f26-istio-envoy\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:27:59.462092 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.461940 5076 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2b862a2-53aa-4c9b-8571-102879f30f26-istio-token\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:27:59.462092 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.461954 5076 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2b862a2-53aa-4c9b-8571-102879f30f26-istiod-ca-cert\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:27:59.462267 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.462238 5076 scope.go:122] "RemoveContainer" containerID="486ee0860be529857341f75bcfbcd69d529966f2da1499c1004c49635cfc3839" Nov 03 00:27:59.470764 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.470738 5076 scope.go:122] "RemoveContainer" containerID="d12c8a1eb27dcabea4246e73494c580e4ac950440607d282ea6008f70af7db6e" Nov 03 00:27:59.471091 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:27:59.471069 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d12c8a1eb27dcabea4246e73494c580e4ac950440607d282ea6008f70af7db6e\": container with ID starting with d12c8a1eb27dcabea4246e73494c580e4ac950440607d282ea6008f70af7db6e not found: ID does not exist" containerID="d12c8a1eb27dcabea4246e73494c580e4ac950440607d282ea6008f70af7db6e" Nov 03 00:27:59.471190 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.471103 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d12c8a1eb27dcabea4246e73494c580e4ac950440607d282ea6008f70af7db6e"} err="failed to get container status \"d12c8a1eb27dcabea4246e73494c580e4ac950440607d282ea6008f70af7db6e\": rpc error: code = NotFound desc = could not find container \"d12c8a1eb27dcabea4246e73494c580e4ac950440607d282ea6008f70af7db6e\": container with ID starting with d12c8a1eb27dcabea4246e73494c580e4ac950440607d282ea6008f70af7db6e not found: ID does not exist" Nov 03 00:27:59.471190 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.471130 5076 scope.go:122] "RemoveContainer" containerID="486ee0860be529857341f75bcfbcd69d529966f2da1499c1004c49635cfc3839" Nov 03 00:27:59.471416 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:27:59.471397 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"486ee0860be529857341f75bcfbcd69d529966f2da1499c1004c49635cfc3839\": container with ID starting with 486ee0860be529857341f75bcfbcd69d529966f2da1499c1004c49635cfc3839 not found: ID does not exist" containerID="486ee0860be529857341f75bcfbcd69d529966f2da1499c1004c49635cfc3839" Nov 03 00:27:59.471462 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.471424 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"486ee0860be529857341f75bcfbcd69d529966f2da1499c1004c49635cfc3839"} err="failed to get container status \"486ee0860be529857341f75bcfbcd69d529966f2da1499c1004c49635cfc3839\": rpc error: code = NotFound desc = could not find container \"486ee0860be529857341f75bcfbcd69d529966f2da1499c1004c49635cfc3839\": container with ID starting with 486ee0860be529857341f75bcfbcd69d529966f2da1499c1004c49635cfc3839 not found: ID does not exist" Nov 03 00:27:59.591807 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.591767 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr"] Nov 03 00:27:59.595788 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:27:59.595761 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-adc00e-openshift-default-58d78f4c49zcslr"] Nov 03 00:28:01.147080 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:28:01.147038 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2b862a2-53aa-4c9b-8571-102879f30f26" path="/var/lib/kubelet/pods/a2b862a2-53aa-4c9b-8571-102879f30f26/volumes" Nov 03 00:28:09.884393 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:28:09.884355 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:28:29.044311 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:28:29.044276 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wx6l5"] Nov 03 00:28:29.091313 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:28:29.091282 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wx6l5"] Nov 03 00:28:29.146660 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:28:29.146630 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0904c42b-d66f-4f78-bf45-51dd67bf969b" path="/var/lib/kubelet/pods/0904c42b-d66f-4f78-bf45-51dd67bf969b/volumes" Nov 03 00:28:33.057488 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:28:33.057459 5076 scope.go:122] "RemoveContainer" containerID="92e5e0fc235f935d97bfa6d166ac9608b0560b16d88f2df1a26da6fa389da219" Nov 03 00:28:39.883034 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:28:39.882996 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:28:46.460369 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:28:46.459987 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gctbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:28:46.460369 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:28:46.460054 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" podUID="d6db6fe8-b76c-4878-b0f3-bd347b2d0cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:28:46.861970 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:28:46.835025 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-784vq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:28:46.861970 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:28:46.835121 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" podUID="506655af-c50a-4207-906e-9f705fc8f9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:28:50.047863 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:28:50.047828 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-84j9c"] Nov 03 00:28:50.095008 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:28:50.094970 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-84j9c"] Nov 03 00:28:51.148737 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:28:51.148703 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69bdd70a-8148-4bac-acf1-4acd28b13f48" path="/var/lib/kubelet/pods/69bdd70a-8148-4bac-acf1-4acd28b13f48/volumes" Nov 03 00:29:02.044269 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:02.044229 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-czjqx"] Nov 03 00:29:02.091600 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:02.091557 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-czjqx"] Nov 03 00:29:03.147183 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:03.147150 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65ab93ca-ad0b-468e-b906-3fae9b5b9be6" path="/var/lib/kubelet/pods/65ab93ca-ad0b-468e-b906-3fae9b5b9be6/volumes" Nov 03 00:29:07.392177 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:07.392140 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2wx9v"] Nov 03 00:29:07.392695 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:07.392599 5076 cpu_manager.go:414] "containerMap: removing container" podUID="a2b862a2-53aa-4c9b-8571-102879f30f26" containerName="istio-proxy" Nov 03 00:29:07.392695 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:07.392611 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2b862a2-53aa-4c9b-8571-102879f30f26" containerName="istio-proxy" Nov 03 00:29:07.392695 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:07.392626 5076 cpu_manager.go:414] "containerMap: removing container" podUID="a2b862a2-53aa-4c9b-8571-102879f30f26" containerName="istio-proxy" Nov 03 00:29:07.392695 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:07.392632 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2b862a2-53aa-4c9b-8571-102879f30f26" containerName="istio-proxy" Nov 03 00:29:07.392695 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:07.392651 5076 cpu_manager.go:414] "containerMap: removing container" podUID="2787345f-875d-4c0d-bf19-0b3417aa5c01" containerName="maas-api" Nov 03 00:29:07.392695 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:07.392656 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2787345f-875d-4c0d-bf19-0b3417aa5c01" containerName="maas-api" Nov 03 00:29:07.392958 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:07.392725 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2b862a2-53aa-4c9b-8571-102879f30f26" containerName="istio-proxy" Nov 03 00:29:07.392958 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:07.392736 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2b862a2-53aa-4c9b-8571-102879f30f26" containerName="istio-proxy" Nov 03 00:29:07.392958 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:07.392745 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="2787345f-875d-4c0d-bf19-0b3417aa5c01" containerName="maas-api" Nov 03 00:29:07.396196 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:07.396180 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2wx9v" Nov 03 00:29:07.451488 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:07.451452 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2wx9v"] Nov 03 00:29:07.457649 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:07.457615 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75e5e93a-6f2e-48bc-9e90-21fd36262cd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-2wx9v\" (UID: \"75e5e93a-6f2e-48bc-9e90-21fd36262cd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2wx9v" Nov 03 00:29:07.457809 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:07.457746 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t478d\" (UniqueName: \"kubernetes.io/projected/75e5e93a-6f2e-48bc-9e90-21fd36262cd0-kube-api-access-t478d\") pod \"maas-api-revoke-sub-452b6d7b-2wx9v\" (UID: \"75e5e93a-6f2e-48bc-9e90-21fd36262cd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2wx9v" Nov 03 00:29:07.457862 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:07.457815 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75e5e93a-6f2e-48bc-9e90-21fd36262cd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-2wx9v\" (UID: \"75e5e93a-6f2e-48bc-9e90-21fd36262cd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2wx9v" Nov 03 00:29:07.495821 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:07.495788 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:29:07.496000 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:07.495823 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:29:07.559138 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:07.559105 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t478d\" (UniqueName: \"kubernetes.io/projected/75e5e93a-6f2e-48bc-9e90-21fd36262cd0-kube-api-access-t478d\") pod \"maas-api-revoke-sub-452b6d7b-2wx9v\" (UID: \"75e5e93a-6f2e-48bc-9e90-21fd36262cd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2wx9v" Nov 03 00:29:07.559312 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:07.559147 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75e5e93a-6f2e-48bc-9e90-21fd36262cd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-2wx9v\" (UID: \"75e5e93a-6f2e-48bc-9e90-21fd36262cd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2wx9v" Nov 03 00:29:07.559312 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:07.559220 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75e5e93a-6f2e-48bc-9e90-21fd36262cd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-2wx9v\" (UID: \"75e5e93a-6f2e-48bc-9e90-21fd36262cd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2wx9v" Nov 03 00:29:07.559839 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:07.559812 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75e5e93a-6f2e-48bc-9e90-21fd36262cd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-2wx9v\" (UID: \"75e5e93a-6f2e-48bc-9e90-21fd36262cd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2wx9v" Nov 03 00:29:07.561493 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:07.561474 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75e5e93a-6f2e-48bc-9e90-21fd36262cd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-2wx9v\" (UID: \"75e5e93a-6f2e-48bc-9e90-21fd36262cd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2wx9v" Nov 03 00:29:07.614184 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:07.614150 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t478d\" (UniqueName: \"kubernetes.io/projected/75e5e93a-6f2e-48bc-9e90-21fd36262cd0-kube-api-access-t478d\") pod \"maas-api-revoke-sub-452b6d7b-2wx9v\" (UID: \"75e5e93a-6f2e-48bc-9e90-21fd36262cd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2wx9v" Nov 03 00:29:07.709230 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:07.709196 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2wx9v" Nov 03 00:29:08.071362 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:08.071327 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2wx9v"] Nov 03 00:29:08.073022 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:29:08.072995 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75e5e93a_6f2e_48bc_9e90_21fd36262cd0.slice/crio-1f4289fbf5024ee4471b587730b2f292bc0deb64465b9ac43d2eb79746746606 WatchSource:0}: Error finding container 1f4289fbf5024ee4471b587730b2f292bc0deb64465b9ac43d2eb79746746606: Status 404 returned error can't find the container with id 1f4289fbf5024ee4471b587730b2f292bc0deb64465b9ac43d2eb79746746606 Nov 03 00:29:08.954199 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:08.954161 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2wx9v" event={"ID":"75e5e93a-6f2e-48bc-9e90-21fd36262cd0","Type":"ContainerStarted","Data":"8cb10fbe2d99767271ce0b4daf7f4f418d2ca33fa4d0d89cdf278ec1230d5d7e"} Nov 03 00:29:08.954596 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:08.954218 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2wx9v" event={"ID":"75e5e93a-6f2e-48bc-9e90-21fd36262cd0","Type":"ContainerStarted","Data":"1f4289fbf5024ee4471b587730b2f292bc0deb64465b9ac43d2eb79746746606"} Nov 03 00:29:09.062966 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:09.062909 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2wx9v" podStartSLOduration=2.062881887 podStartE2EDuration="2.062881887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:29:09.062097395 +0000 UTC m=+5798.030629180" watchObservedRunningTime="2026-11-03 00:29:09.062881887 +0000 UTC m=+5798.031413670" Nov 03 00:29:09.896831 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:29:09.896785 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:29:09.965451 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:09.965411 5076 generic.go:356] "Generic (PLEG): container finished" podID="75e5e93a-6f2e-48bc-9e90-21fd36262cd0" containerID="8cb10fbe2d99767271ce0b4daf7f4f418d2ca33fa4d0d89cdf278ec1230d5d7e" exitCode=0 Nov 03 00:29:09.966095 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:09.965551 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2wx9v" event={"ID":"75e5e93a-6f2e-48bc-9e90-21fd36262cd0","Type":"ContainerDied","Data":"8cb10fbe2d99767271ce0b4daf7f4f418d2ca33fa4d0d89cdf278ec1230d5d7e"} Nov 03 00:29:12.134364 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:12.134328 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2wx9v" Nov 03 00:29:12.159501 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:12.159393 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2wx9v" event={"ID":"75e5e93a-6f2e-48bc-9e90-21fd36262cd0","Type":"ContainerDied","Data":"1f4289fbf5024ee4471b587730b2f292bc0deb64465b9ac43d2eb79746746606"} Nov 03 00:29:12.159809 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:12.159769 5076 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f4289fbf5024ee4471b587730b2f292bc0deb64465b9ac43d2eb79746746606" Nov 03 00:29:12.297594 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:12.296309 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75e5e93a-6f2e-48bc-9e90-21fd36262cd0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75e5e93a-6f2e-48bc-9e90-21fd36262cd0-maas-api-xks-ca\") pod \"75e5e93a-6f2e-48bc-9e90-21fd36262cd0\" (UID: \"75e5e93a-6f2e-48bc-9e90-21fd36262cd0\") " Nov 03 00:29:12.297594 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:12.296404 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75e5e93a-6f2e-48bc-9e90-21fd36262cd0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75e5e93a-6f2e-48bc-9e90-21fd36262cd0-maas-api-openshift-ca\") pod \"75e5e93a-6f2e-48bc-9e90-21fd36262cd0\" (UID: \"75e5e93a-6f2e-48bc-9e90-21fd36262cd0\") " Nov 03 00:29:12.297594 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:12.296506 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75e5e93a-6f2e-48bc-9e90-21fd36262cd0-kube-api-access-t478d\" (UniqueName: \"kubernetes.io/projected/75e5e93a-6f2e-48bc-9e90-21fd36262cd0-kube-api-access-t478d\") pod \"75e5e93a-6f2e-48bc-9e90-21fd36262cd0\" (UID: \"75e5e93a-6f2e-48bc-9e90-21fd36262cd0\") " Nov 03 00:29:12.310195 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:12.310158 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75e5e93a-6f2e-48bc-9e90-21fd36262cd0-maas-api-openshift-ca" pod "75e5e93a-6f2e-48bc-9e90-21fd36262cd0" (UID: "75e5e93a-6f2e-48bc-9e90-21fd36262cd0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:29:12.369159 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:12.369083 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75e5e93a-6f2e-48bc-9e90-21fd36262cd0-kube-api-access-t478d" pod "75e5e93a-6f2e-48bc-9e90-21fd36262cd0" (UID: "75e5e93a-6f2e-48bc-9e90-21fd36262cd0"). InnerVolumeSpecName "kube-api-access-t478d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:29:12.421246 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:12.421099 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75e5e93a-6f2e-48bc-9e90-21fd36262cd0-maas-api-xks-ca" pod "75e5e93a-6f2e-48bc-9e90-21fd36262cd0" (UID: "75e5e93a-6f2e-48bc-9e90-21fd36262cd0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:29:12.487916 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:12.449607 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75e5e93a-6f2e-48bc-9e90-21fd36262cd0-maas-api-xks-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:29:12.493918 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:12.488248 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75e5e93a-6f2e-48bc-9e90-21fd36262cd0-maas-api-openshift-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:29:12.502912 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:12.494155 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t478d\" (UniqueName: \"kubernetes.io/projected/75e5e93a-6f2e-48bc-9e90-21fd36262cd0-kube-api-access-t478d\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:29:13.165538 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:13.165509 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2wx9v" Nov 03 00:29:27.342926 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:27.342874 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zhks7"] Nov 03 00:29:27.343635 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:27.343613 5076 cpu_manager.go:414] "containerMap: removing container" podUID="75e5e93a-6f2e-48bc-9e90-21fd36262cd0" containerName="revoke-keys" Nov 03 00:29:27.343710 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:27.343638 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75e5e93a-6f2e-48bc-9e90-21fd36262cd0" containerName="revoke-keys" Nov 03 00:29:27.343834 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:27.343821 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="75e5e93a-6f2e-48bc-9e90-21fd36262cd0" containerName="revoke-keys" Nov 03 00:29:27.347523 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:27.347490 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zhks7" Nov 03 00:29:27.399944 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:27.399881 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zhks7"] Nov 03 00:29:27.424300 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:27.424268 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c183247-b2f3-4f68-afa2-233a8a8bb647-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-zhks7\" (UID: \"0c183247-b2f3-4f68-afa2-233a8a8bb647\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zhks7" Nov 03 00:29:27.424471 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:27.424316 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c183247-b2f3-4f68-afa2-233a8a8bb647-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-zhks7\" (UID: \"0c183247-b2f3-4f68-afa2-233a8a8bb647\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zhks7" Nov 03 00:29:27.424471 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:27.424427 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmqlh\" (UniqueName: \"kubernetes.io/projected/0c183247-b2f3-4f68-afa2-233a8a8bb647-kube-api-access-bmqlh\") pod \"maas-api-revoke-sub-3e31a506-zhks7\" (UID: \"0c183247-b2f3-4f68-afa2-233a8a8bb647\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zhks7" Nov 03 00:29:27.444272 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:27.444245 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:29:27.444272 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:27.444247 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:29:27.525691 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:27.525660 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c183247-b2f3-4f68-afa2-233a8a8bb647-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-zhks7\" (UID: \"0c183247-b2f3-4f68-afa2-233a8a8bb647\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zhks7" Nov 03 00:29:27.525915 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:27.525701 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c183247-b2f3-4f68-afa2-233a8a8bb647-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-zhks7\" (UID: \"0c183247-b2f3-4f68-afa2-233a8a8bb647\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zhks7" Nov 03 00:29:27.525915 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:27.525750 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmqlh\" (UniqueName: \"kubernetes.io/projected/0c183247-b2f3-4f68-afa2-233a8a8bb647-kube-api-access-bmqlh\") pod \"maas-api-revoke-sub-3e31a506-zhks7\" (UID: \"0c183247-b2f3-4f68-afa2-233a8a8bb647\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zhks7" Nov 03 00:29:27.526353 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:27.526329 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c183247-b2f3-4f68-afa2-233a8a8bb647-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-zhks7\" (UID: \"0c183247-b2f3-4f68-afa2-233a8a8bb647\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zhks7" Nov 03 00:29:27.528055 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:27.528035 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c183247-b2f3-4f68-afa2-233a8a8bb647-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-zhks7\" (UID: \"0c183247-b2f3-4f68-afa2-233a8a8bb647\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zhks7" Nov 03 00:29:27.580390 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:27.580363 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmqlh\" (UniqueName: \"kubernetes.io/projected/0c183247-b2f3-4f68-afa2-233a8a8bb647-kube-api-access-bmqlh\") pod \"maas-api-revoke-sub-3e31a506-zhks7\" (UID: \"0c183247-b2f3-4f68-afa2-233a8a8bb647\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zhks7" Nov 03 00:29:27.660497 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:27.660405 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zhks7" Nov 03 00:29:28.026293 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:28.026261 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zhks7"] Nov 03 00:29:28.027730 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:29:28.027700 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c183247_b2f3_4f68_afa2_233a8a8bb647.slice/crio-fa00f6a2f066e3070b8aac5d166d111141a79089f36d9e84644f97c5a5a1457c WatchSource:0}: Error finding container fa00f6a2f066e3070b8aac5d166d111141a79089f36d9e84644f97c5a5a1457c: Status 404 returned error can't find the container with id fa00f6a2f066e3070b8aac5d166d111141a79089f36d9e84644f97c5a5a1457c Nov 03 00:29:28.291628 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:28.291545 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zhks7" event={"ID":"0c183247-b2f3-4f68-afa2-233a8a8bb647","Type":"ContainerStarted","Data":"fa00f6a2f066e3070b8aac5d166d111141a79089f36d9e84644f97c5a5a1457c"} Nov 03 00:29:29.298187 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:29.298155 5076 generic.go:356] "Generic (PLEG): container finished" podID="0c183247-b2f3-4f68-afa2-233a8a8bb647" containerID="799763e3f06cbfd266fa9cd4b915a61413b1d6502ffe675b103ae80404910cbc" exitCode=0 Nov 03 00:29:29.298617 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:29.298246 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zhks7" event={"ID":"0c183247-b2f3-4f68-afa2-233a8a8bb647","Type":"ContainerDied","Data":"799763e3f06cbfd266fa9cd4b915a61413b1d6502ffe675b103ae80404910cbc"} Nov 03 00:29:30.509726 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:30.509678 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zhks7" Nov 03 00:29:30.557350 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:30.557219 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c183247-b2f3-4f68-afa2-233a8a8bb647-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c183247-b2f3-4f68-afa2-233a8a8bb647-maas-api-openshift-ca\") pod \"0c183247-b2f3-4f68-afa2-233a8a8bb647\" (UID: \"0c183247-b2f3-4f68-afa2-233a8a8bb647\") " Nov 03 00:29:30.558048 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:30.557443 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c183247-b2f3-4f68-afa2-233a8a8bb647-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c183247-b2f3-4f68-afa2-233a8a8bb647-maas-api-xks-ca\") pod \"0c183247-b2f3-4f68-afa2-233a8a8bb647\" (UID: \"0c183247-b2f3-4f68-afa2-233a8a8bb647\") " Nov 03 00:29:30.558048 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:30.557488 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c183247-b2f3-4f68-afa2-233a8a8bb647-kube-api-access-bmqlh\" (UniqueName: \"kubernetes.io/projected/0c183247-b2f3-4f68-afa2-233a8a8bb647-kube-api-access-bmqlh\") pod \"0c183247-b2f3-4f68-afa2-233a8a8bb647\" (UID: \"0c183247-b2f3-4f68-afa2-233a8a8bb647\") " Nov 03 00:29:30.558048 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:30.557658 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c183247-b2f3-4f68-afa2-233a8a8bb647-maas-api-openshift-ca" pod "0c183247-b2f3-4f68-afa2-233a8a8bb647" (UID: "0c183247-b2f3-4f68-afa2-233a8a8bb647"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:29:30.558048 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:30.558005 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c183247-b2f3-4f68-afa2-233a8a8bb647-maas-api-openshift-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:29:30.560323 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:30.560155 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c183247-b2f3-4f68-afa2-233a8a8bb647-maas-api-xks-ca" pod "0c183247-b2f3-4f68-afa2-233a8a8bb647" (UID: "0c183247-b2f3-4f68-afa2-233a8a8bb647"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:29:30.560596 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:30.560563 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c183247-b2f3-4f68-afa2-233a8a8bb647-kube-api-access-bmqlh" pod "0c183247-b2f3-4f68-afa2-233a8a8bb647" (UID: "0c183247-b2f3-4f68-afa2-233a8a8bb647"). InnerVolumeSpecName "kube-api-access-bmqlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:29:30.658867 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:30.658826 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c183247-b2f3-4f68-afa2-233a8a8bb647-maas-api-xks-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:29:30.658867 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:30.658866 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmqlh\" (UniqueName: \"kubernetes.io/projected/0c183247-b2f3-4f68-afa2-233a8a8bb647-kube-api-access-bmqlh\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:29:31.313335 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:31.313301 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zhks7" Nov 03 00:29:31.313545 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:31.313297 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zhks7" event={"ID":"0c183247-b2f3-4f68-afa2-233a8a8bb647","Type":"ContainerDied","Data":"fa00f6a2f066e3070b8aac5d166d111141a79089f36d9e84644f97c5a5a1457c"} Nov 03 00:29:31.313545 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:31.313443 5076 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa00f6a2f066e3070b8aac5d166d111141a79089f36d9e84644f97c5a5a1457c" Nov 03 00:29:33.085918 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:33.085380 5076 scope.go:122] "RemoveContainer" containerID="7b23d5b7e6d69d4c86d65f1ea68cc2c2f2e048d24efab6ca9a71adcc063e385d" Nov 03 00:29:33.102227 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:33.101834 5076 scope.go:122] "RemoveContainer" containerID="65e60aa59f6eeb37a45a30aec23e7846347a93b0bdeda64ea956d150d3db2c51" Nov 03 00:29:39.882651 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:29:39.882609 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:29:51.305928 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:51.305877 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83a4f24f-4ppm6"] Nov 03 00:29:51.306352 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:51.306338 5076 cpu_manager.go:414] "containerMap: removing container" podUID="0c183247-b2f3-4f68-afa2-233a8a8bb647" containerName="revoke-keys" Nov 03 00:29:51.306398 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:51.306352 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c183247-b2f3-4f68-afa2-233a8a8bb647" containerName="revoke-keys" Nov 03 00:29:51.306438 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:51.306424 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c183247-b2f3-4f68-afa2-233a8a8bb647" containerName="revoke-keys" Nov 03 00:29:51.309146 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:51.309120 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a4f24f-4ppm6" Nov 03 00:29:51.366403 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:51.366367 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83a4f24f-4ppm6"] Nov 03 00:29:51.409185 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:51.409157 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:29:51.409185 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:51.409156 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:29:51.411675 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:51.411654 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79069e48-269e-42bc-a59a-a71e9c1aea4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83a4f24f-4ppm6\" (UID: \"79069e48-269e-42bc-a59a-a71e9c1aea4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a4f24f-4ppm6" Nov 03 00:29:51.411732 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:51.411697 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndrpt\" (UniqueName: \"kubernetes.io/projected/79069e48-269e-42bc-a59a-a71e9c1aea4a-kube-api-access-ndrpt\") pod \"maas-api-revoke-sub-83a4f24f-4ppm6\" (UID: \"79069e48-269e-42bc-a59a-a71e9c1aea4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a4f24f-4ppm6" Nov 03 00:29:51.411795 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:51.411773 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79069e48-269e-42bc-a59a-a71e9c1aea4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83a4f24f-4ppm6\" (UID: \"79069e48-269e-42bc-a59a-a71e9c1aea4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a4f24f-4ppm6" Nov 03 00:29:51.512597 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:51.512562 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79069e48-269e-42bc-a59a-a71e9c1aea4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83a4f24f-4ppm6\" (UID: \"79069e48-269e-42bc-a59a-a71e9c1aea4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a4f24f-4ppm6" Nov 03 00:29:51.512808 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:51.512772 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79069e48-269e-42bc-a59a-a71e9c1aea4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83a4f24f-4ppm6\" (UID: \"79069e48-269e-42bc-a59a-a71e9c1aea4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a4f24f-4ppm6" Nov 03 00:29:51.512808 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:51.512808 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndrpt\" (UniqueName: \"kubernetes.io/projected/79069e48-269e-42bc-a59a-a71e9c1aea4a-kube-api-access-ndrpt\") pod \"maas-api-revoke-sub-83a4f24f-4ppm6\" (UID: \"79069e48-269e-42bc-a59a-a71e9c1aea4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a4f24f-4ppm6" Nov 03 00:29:51.513515 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:51.513493 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79069e48-269e-42bc-a59a-a71e9c1aea4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83a4f24f-4ppm6\" (UID: \"79069e48-269e-42bc-a59a-a71e9c1aea4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a4f24f-4ppm6" Nov 03 00:29:51.515055 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:51.515034 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79069e48-269e-42bc-a59a-a71e9c1aea4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83a4f24f-4ppm6\" (UID: \"79069e48-269e-42bc-a59a-a71e9c1aea4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a4f24f-4ppm6" Nov 03 00:29:51.567084 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:51.567012 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndrpt\" (UniqueName: \"kubernetes.io/projected/79069e48-269e-42bc-a59a-a71e9c1aea4a-kube-api-access-ndrpt\") pod \"maas-api-revoke-sub-83a4f24f-4ppm6\" (UID: \"79069e48-269e-42bc-a59a-a71e9c1aea4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a4f24f-4ppm6" Nov 03 00:29:51.621323 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:51.621288 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a4f24f-4ppm6" Nov 03 00:29:51.988991 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:51.988959 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83a4f24f-4ppm6"] Nov 03 00:29:51.990426 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:29:51.990398 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79069e48_269e_42bc_a59a_a71e9c1aea4a.slice/crio-a143904f2a7dec6f99f422281e907355ec1ac0d9f0601b2923c5fe31ea4a786d WatchSource:0}: Error finding container a143904f2a7dec6f99f422281e907355ec1ac0d9f0601b2923c5fe31ea4a786d: Status 404 returned error can't find the container with id a143904f2a7dec6f99f422281e907355ec1ac0d9f0601b2923c5fe31ea4a786d Nov 03 00:29:52.513843 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:52.513809 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a4f24f-4ppm6" event={"ID":"79069e48-269e-42bc-a59a-a71e9c1aea4a","Type":"ContainerStarted","Data":"8cf0c1f898acf597eb56e74847955de4f7905639b1aa8ed7a5d361db5647808c"} Nov 03 00:29:52.514317 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:52.513864 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a4f24f-4ppm6" event={"ID":"79069e48-269e-42bc-a59a-a71e9c1aea4a","Type":"ContainerStarted","Data":"a143904f2a7dec6f99f422281e907355ec1ac0d9f0601b2923c5fe31ea4a786d"} Nov 03 00:29:52.623406 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:52.623281 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a4f24f-4ppm6" podStartSLOduration=1.623264335 podStartE2EDuration="1.623264335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:29:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:29:52.622517881 +0000 UTC m=+5841.591049666" watchObservedRunningTime="2026-11-03 00:29:52.623264335 +0000 UTC m=+5841.591796119" Nov 03 00:29:53.520974 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:53.520941 5076 generic.go:356] "Generic (PLEG): container finished" podID="79069e48-269e-42bc-a59a-a71e9c1aea4a" containerID="8cf0c1f898acf597eb56e74847955de4f7905639b1aa8ed7a5d361db5647808c" exitCode=0 Nov 03 00:29:53.521392 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:53.521037 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a4f24f-4ppm6" event={"ID":"79069e48-269e-42bc-a59a-a71e9c1aea4a","Type":"ContainerDied","Data":"8cf0c1f898acf597eb56e74847955de4f7905639b1aa8ed7a5d361db5647808c"} Nov 03 00:29:54.676226 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:54.676199 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a4f24f-4ppm6" Nov 03 00:29:54.745440 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:54.745406 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79069e48-269e-42bc-a59a-a71e9c1aea4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79069e48-269e-42bc-a59a-a71e9c1aea4a-maas-api-openshift-ca\") pod \"79069e48-269e-42bc-a59a-a71e9c1aea4a\" (UID: \"79069e48-269e-42bc-a59a-a71e9c1aea4a\") " Nov 03 00:29:54.745635 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:54.745461 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79069e48-269e-42bc-a59a-a71e9c1aea4a-kube-api-access-ndrpt\" (UniqueName: \"kubernetes.io/projected/79069e48-269e-42bc-a59a-a71e9c1aea4a-kube-api-access-ndrpt\") pod \"79069e48-269e-42bc-a59a-a71e9c1aea4a\" (UID: \"79069e48-269e-42bc-a59a-a71e9c1aea4a\") " Nov 03 00:29:54.745635 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:54.745544 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79069e48-269e-42bc-a59a-a71e9c1aea4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79069e48-269e-42bc-a59a-a71e9c1aea4a-maas-api-xks-ca\") pod \"79069e48-269e-42bc-a59a-a71e9c1aea4a\" (UID: \"79069e48-269e-42bc-a59a-a71e9c1aea4a\") " Nov 03 00:29:54.745845 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:54.745815 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79069e48-269e-42bc-a59a-a71e9c1aea4a-maas-api-openshift-ca" pod "79069e48-269e-42bc-a59a-a71e9c1aea4a" (UID: "79069e48-269e-42bc-a59a-a71e9c1aea4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:29:54.747677 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:54.747649 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79069e48-269e-42bc-a59a-a71e9c1aea4a-maas-api-xks-ca" pod "79069e48-269e-42bc-a59a-a71e9c1aea4a" (UID: "79069e48-269e-42bc-a59a-a71e9c1aea4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:29:54.747787 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:54.747759 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79069e48-269e-42bc-a59a-a71e9c1aea4a-kube-api-access-ndrpt" pod "79069e48-269e-42bc-a59a-a71e9c1aea4a" (UID: "79069e48-269e-42bc-a59a-a71e9c1aea4a"). InnerVolumeSpecName "kube-api-access-ndrpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:29:54.846793 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:54.846716 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79069e48-269e-42bc-a59a-a71e9c1aea4a-maas-api-xks-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:29:54.846793 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:54.846743 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79069e48-269e-42bc-a59a-a71e9c1aea4a-maas-api-openshift-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:29:54.846793 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:54.846754 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndrpt\" (UniqueName: \"kubernetes.io/projected/79069e48-269e-42bc-a59a-a71e9c1aea4a-kube-api-access-ndrpt\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:29:55.532751 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:55.532719 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a4f24f-4ppm6" Nov 03 00:29:55.532957 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:55.532749 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a4f24f-4ppm6" event={"ID":"79069e48-269e-42bc-a59a-a71e9c1aea4a","Type":"ContainerDied","Data":"a143904f2a7dec6f99f422281e907355ec1ac0d9f0601b2923c5fe31ea4a786d"} Nov 03 00:29:55.532957 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:29:55.532800 5076 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a143904f2a7dec6f99f422281e907355ec1ac0d9f0601b2923c5fe31ea4a786d" Nov 03 00:30:00.175044 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:00.174998 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894430-rnv45"] Nov 03 00:30:00.175572 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:00.175498 5076 cpu_manager.go:414] "containerMap: removing container" podUID="79069e48-269e-42bc-a59a-a71e9c1aea4a" containerName="revoke-keys" Nov 03 00:30:00.175572 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:00.175513 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79069e48-269e-42bc-a59a-a71e9c1aea4a" containerName="revoke-keys" Nov 03 00:30:00.175685 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:00.175639 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="79069e48-269e-42bc-a59a-a71e9c1aea4a" containerName="revoke-keys" Nov 03 00:30:00.179160 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:00.179140 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-rnv45" Nov 03 00:30:00.229958 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:00.229927 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894430-rnv45"] Nov 03 00:30:00.274338 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:00.274304 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:30:00.299113 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:00.299076 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b84804de-71f5-4e83-84b2-68ad779ea6ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894430-rnv45\" (UID: \"b84804de-71f5-4e83-84b2-68ad779ea6ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-rnv45" Nov 03 00:30:00.299300 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:00.299123 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dvbb\" (UniqueName: \"kubernetes.io/projected/b84804de-71f5-4e83-84b2-68ad779ea6ea-kube-api-access-8dvbb\") pod \"maas-api-key-cleanup-29894430-rnv45\" (UID: \"b84804de-71f5-4e83-84b2-68ad779ea6ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-rnv45" Nov 03 00:30:00.399676 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:00.399638 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b84804de-71f5-4e83-84b2-68ad779ea6ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894430-rnv45\" (UID: \"b84804de-71f5-4e83-84b2-68ad779ea6ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-rnv45" Nov 03 00:30:00.399676 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:00.399680 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dvbb\" (UniqueName: \"kubernetes.io/projected/b84804de-71f5-4e83-84b2-68ad779ea6ea-kube-api-access-8dvbb\") pod \"maas-api-key-cleanup-29894430-rnv45\" (UID: \"b84804de-71f5-4e83-84b2-68ad779ea6ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-rnv45" Nov 03 00:30:00.400307 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:00.400285 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b84804de-71f5-4e83-84b2-68ad779ea6ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894430-rnv45\" (UID: \"b84804de-71f5-4e83-84b2-68ad779ea6ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-rnv45" Nov 03 00:30:00.456936 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:00.456883 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dvbb\" (UniqueName: \"kubernetes.io/projected/b84804de-71f5-4e83-84b2-68ad779ea6ea-kube-api-access-8dvbb\") pod \"maas-api-key-cleanup-29894430-rnv45\" (UID: \"b84804de-71f5-4e83-84b2-68ad779ea6ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-rnv45" Nov 03 00:30:00.492526 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:00.492500 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-rnv45" Nov 03 00:30:00.858460 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:00.858429 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894430-rnv45"] Nov 03 00:30:00.859979 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:30:00.859948 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb84804de_71f5_4e83_84b2_68ad779ea6ea.slice/crio-313adaba0febc3036c905c8df0706da5e069fa1716e5e49ef3402b078a6de0c0 WatchSource:0}: Error finding container 313adaba0febc3036c905c8df0706da5e069fa1716e5e49ef3402b078a6de0c0: Status 404 returned error can't find the container with id 313adaba0febc3036c905c8df0706da5e069fa1716e5e49ef3402b078a6de0c0 Nov 03 00:30:01.573035 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:01.573005 5076 generic.go:356] "Generic (PLEG): container finished" podID="b84804de-71f5-4e83-84b2-68ad779ea6ea" containerID="4a805abb8e82ceeb36ddb0237a1749643c8ffa91b061c608c435caf721faafe5" exitCode=60 Nov 03 00:30:01.573448 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:01.573090 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-rnv45" event={"ID":"b84804de-71f5-4e83-84b2-68ad779ea6ea","Type":"ContainerDied","Data":"4a805abb8e82ceeb36ddb0237a1749643c8ffa91b061c608c435caf721faafe5"} Nov 03 00:30:01.573448 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:01.573160 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-rnv45" event={"ID":"b84804de-71f5-4e83-84b2-68ad779ea6ea","Type":"ContainerStarted","Data":"313adaba0febc3036c905c8df0706da5e069fa1716e5e49ef3402b078a6de0c0"} Nov 03 00:30:01.573448 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:01.573300 5076 scope.go:122] "RemoveContainer" containerID="4a805abb8e82ceeb36ddb0237a1749643c8ffa91b061c608c435caf721faafe5" Nov 03 00:30:02.580694 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:02.580658 5076 generic.go:356] "Generic (PLEG): container finished" podID="b84804de-71f5-4e83-84b2-68ad779ea6ea" containerID="3ec57eb5dcae5ae5d61738b6475e17576044152cbcfba32b8ae95b9a42533a10" exitCode=60 Nov 03 00:30:02.581149 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:02.580738 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-rnv45" event={"ID":"b84804de-71f5-4e83-84b2-68ad779ea6ea","Type":"ContainerDied","Data":"3ec57eb5dcae5ae5d61738b6475e17576044152cbcfba32b8ae95b9a42533a10"} Nov 03 00:30:02.581149 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:02.580796 5076 scope.go:122] "RemoveContainer" containerID="4a805abb8e82ceeb36ddb0237a1749643c8ffa91b061c608c435caf721faafe5" Nov 03 00:30:02.581149 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:02.581091 5076 scope.go:122] "RemoveContainer" containerID="3ec57eb5dcae5ae5d61738b6475e17576044152cbcfba32b8ae95b9a42533a10" Nov 03 00:30:02.581399 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:30:02.581377 5076 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894430-rnv45_odh-ai-gateway-infra(b84804de-71f5-4e83-84b2-68ad779ea6ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-rnv45" podUID="b84804de-71f5-4e83-84b2-68ad779ea6ea" Nov 03 00:30:03.587572 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:03.587545 5076 scope.go:122] "RemoveContainer" containerID="3ec57eb5dcae5ae5d61738b6475e17576044152cbcfba32b8ae95b9a42533a10" Nov 03 00:30:03.588056 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:30:03.587792 5076 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894430-rnv45_odh-ai-gateway-infra(b84804de-71f5-4e83-84b2-68ad779ea6ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-rnv45" podUID="b84804de-71f5-4e83-84b2-68ad779ea6ea" Nov 03 00:30:09.883553 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:30:09.883502 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:30:12.045322 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:12.045282 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2c9e7c5-w6wjx"] Nov 03 00:30:12.092474 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:12.092431 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2c9e7c5-w6wjx"] Nov 03 00:30:13.147005 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:13.146971 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="185823d3-c448-42d6-ba42-fb7f86ce7cc0" path="/var/lib/kubelet/pods/185823d3-c448-42d6-ba42-fb7f86ce7cc0/volumes" Nov 03 00:30:15.147471 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:15.147441 5076 scope.go:122] "RemoveContainer" containerID="3ec57eb5dcae5ae5d61738b6475e17576044152cbcfba32b8ae95b9a42533a10" Nov 03 00:30:15.658212 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:15.658178 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-rnv45" event={"ID":"b84804de-71f5-4e83-84b2-68ad779ea6ea","Type":"ContainerStarted","Data":"83445c571862166de60f79201e3d83a3796e76890443583e0dea0f2dc7fb302d"} Nov 03 00:30:15.768807 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:15.768756 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-rnv45" podStartSLOduration=15.768741018 podStartE2EDuration="15.768741018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:15.76701883 +0000 UTC m=+5864.735550616" watchObservedRunningTime="2026-11-03 00:30:15.768741018 +0000 UTC m=+5864.737272852" Nov 03 00:30:16.665159 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:16.665126 5076 generic.go:356] "Generic (PLEG): container finished" podID="b84804de-71f5-4e83-84b2-68ad779ea6ea" containerID="83445c571862166de60f79201e3d83a3796e76890443583e0dea0f2dc7fb302d" exitCode=60 Nov 03 00:30:16.665634 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:16.665196 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-rnv45" event={"ID":"b84804de-71f5-4e83-84b2-68ad779ea6ea","Type":"ContainerDied","Data":"83445c571862166de60f79201e3d83a3796e76890443583e0dea0f2dc7fb302d"} Nov 03 00:30:16.665634 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:16.665255 5076 scope.go:122] "RemoveContainer" containerID="3ec57eb5dcae5ae5d61738b6475e17576044152cbcfba32b8ae95b9a42533a10" Nov 03 00:30:16.665634 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:16.665623 5076 scope.go:122] "RemoveContainer" containerID="83445c571862166de60f79201e3d83a3796e76890443583e0dea0f2dc7fb302d" Nov 03 00:30:16.665939 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:30:16.665912 5076 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894430-rnv45_odh-ai-gateway-infra(b84804de-71f5-4e83-84b2-68ad779ea6ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-rnv45" podUID="b84804de-71f5-4e83-84b2-68ad779ea6ea" Nov 03 00:30:16.783878 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:16.783846 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894430-rnv45"] Nov 03 00:30:17.815456 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:17.815430 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-rnv45" Nov 03 00:30:17.880299 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:17.880258 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b84804de-71f5-4e83-84b2-68ad779ea6ea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b84804de-71f5-4e83-84b2-68ad779ea6ea-maas-api-service-ca\") pod \"b84804de-71f5-4e83-84b2-68ad779ea6ea\" (UID: \"b84804de-71f5-4e83-84b2-68ad779ea6ea\") " Nov 03 00:30:17.880502 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:17.880309 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b84804de-71f5-4e83-84b2-68ad779ea6ea-kube-api-access-8dvbb\" (UniqueName: \"kubernetes.io/projected/b84804de-71f5-4e83-84b2-68ad779ea6ea-kube-api-access-8dvbb\") pod \"b84804de-71f5-4e83-84b2-68ad779ea6ea\" (UID: \"b84804de-71f5-4e83-84b2-68ad779ea6ea\") " Nov 03 00:30:17.880590 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:17.880565 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b84804de-71f5-4e83-84b2-68ad779ea6ea-maas-api-service-ca" pod "b84804de-71f5-4e83-84b2-68ad779ea6ea" (UID: "b84804de-71f5-4e83-84b2-68ad779ea6ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:30:17.880778 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:17.880765 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b84804de-71f5-4e83-84b2-68ad779ea6ea-maas-api-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:30:17.882359 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:17.882340 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b84804de-71f5-4e83-84b2-68ad779ea6ea-kube-api-access-8dvbb" pod "b84804de-71f5-4e83-84b2-68ad779ea6ea" (UID: "b84804de-71f5-4e83-84b2-68ad779ea6ea"). InnerVolumeSpecName "kube-api-access-8dvbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:30:17.982136 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:17.982104 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dvbb\" (UniqueName: \"kubernetes.io/projected/b84804de-71f5-4e83-84b2-68ad779ea6ea-kube-api-access-8dvbb\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:30:18.678805 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:18.678779 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-rnv45" Nov 03 00:30:18.679007 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:18.678771 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-rnv45" event={"ID":"b84804de-71f5-4e83-84b2-68ad779ea6ea","Type":"ContainerDied","Data":"313adaba0febc3036c905c8df0706da5e069fa1716e5e49ef3402b078a6de0c0"} Nov 03 00:30:18.679007 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:18.678924 5076 scope.go:122] "RemoveContainer" containerID="83445c571862166de60f79201e3d83a3796e76890443583e0dea0f2dc7fb302d" Nov 03 00:30:18.838288 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:18.838251 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894430-rnv45"] Nov 03 00:30:18.886124 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:18.886091 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894430-rnv45"] Nov 03 00:30:19.147186 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:19.147142 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b84804de-71f5-4e83-84b2-68ad779ea6ea" path="/var/lib/kubelet/pods/b84804de-71f5-4e83-84b2-68ad779ea6ea/volumes" Nov 03 00:30:31.045248 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:31.045213 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fffe4bab-mcqm6"] Nov 03 00:30:31.092166 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:31.092129 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fffe4bab-mcqm6"] Nov 03 00:30:31.146911 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:31.146859 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="421ba8d5-6602-4aef-bcb5-9518f1843d33" path="/var/lib/kubelet/pods/421ba8d5-6602-4aef-bcb5-9518f1843d33/volumes" Nov 03 00:30:33.246787 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:33.246755 5076 scope.go:122] "RemoveContainer" containerID="9a0621e72819c1a1123df364e0bca8560d9c812e94753359b75b980c652c8e66" Nov 03 00:30:33.255687 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:30:33.255665 5076 scope.go:122] "RemoveContainer" containerID="7ff63cef444bcdb432517a537218f67999b8edeed2a8b807bf0b478b085f6605" Nov 03 00:30:39.883218 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:30:39.883179 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:31:09.882642 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:31:09.882598 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:31:33.283731 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:31:33.283698 5076 scope.go:122] "RemoveContainer" containerID="354a60448a79e26f3c78e7792d99c1e9139fce4b8353e644da52bb0a4c2af921" Nov 03 00:31:33.292695 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:31:33.292674 5076 scope.go:122] "RemoveContainer" containerID="bcee262a98f90f2d989294ef4a2337a6e8c7981b6cce065f732c86bbb937c3bf" Nov 03 00:31:39.882832 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:31:39.882789 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:32:09.884090 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:32:09.884052 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:32:32.287502 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:32.287427 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 03 00:32:32.387640 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:32.387614 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 03 00:32:37.988030 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:37.987997 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wqw5q"] Nov 03 00:32:37.988438 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:37.988424 5076 cpu_manager.go:414] "containerMap: removing container" podUID="b84804de-71f5-4e83-84b2-68ad779ea6ea" containerName="cleanup" Nov 03 00:32:37.988483 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:37.988439 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b84804de-71f5-4e83-84b2-68ad779ea6ea" containerName="cleanup" Nov 03 00:32:37.988483 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:37.988452 5076 cpu_manager.go:414] "containerMap: removing container" podUID="b84804de-71f5-4e83-84b2-68ad779ea6ea" containerName="cleanup" Nov 03 00:32:37.988483 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:37.988457 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b84804de-71f5-4e83-84b2-68ad779ea6ea" containerName="cleanup" Nov 03 00:32:37.988483 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:37.988467 5076 cpu_manager.go:414] "containerMap: removing container" podUID="b84804de-71f5-4e83-84b2-68ad779ea6ea" containerName="cleanup" Nov 03 00:32:37.988483 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:37.988472 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b84804de-71f5-4e83-84b2-68ad779ea6ea" containerName="cleanup" Nov 03 00:32:37.988644 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:37.988555 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="b84804de-71f5-4e83-84b2-68ad779ea6ea" containerName="cleanup" Nov 03 00:32:37.988644 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:37.988564 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="b84804de-71f5-4e83-84b2-68ad779ea6ea" containerName="cleanup" Nov 03 00:32:37.988644 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:37.988570 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="b84804de-71f5-4e83-84b2-68ad779ea6ea" containerName="cleanup" Nov 03 00:32:37.991714 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:37.991698 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wqw5q" Nov 03 00:32:38.044595 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:38.044565 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wqw5q"] Nov 03 00:32:38.070848 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:38.070805 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zslr4\" (UniqueName: \"kubernetes.io/projected/5e3a97b9-b122-48e9-97c4-15b1a0e43e26-kube-api-access-zslr4\") pod \"maas-api-revoke-sub-e2547622-wqw5q\" (UID: \"5e3a97b9-b122-48e9-97c4-15b1a0e43e26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wqw5q" Nov 03 00:32:38.071046 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:38.071023 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e3a97b9-b122-48e9-97c4-15b1a0e43e26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-wqw5q\" (UID: \"5e3a97b9-b122-48e9-97c4-15b1a0e43e26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wqw5q" Nov 03 00:32:38.071093 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:38.071066 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e3a97b9-b122-48e9-97c4-15b1a0e43e26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-wqw5q\" (UID: \"5e3a97b9-b122-48e9-97c4-15b1a0e43e26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wqw5q" Nov 03 00:32:38.089320 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:38.089292 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:32:38.089485 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:38.089427 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:32:38.171637 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:38.171597 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e3a97b9-b122-48e9-97c4-15b1a0e43e26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-wqw5q\" (UID: \"5e3a97b9-b122-48e9-97c4-15b1a0e43e26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wqw5q" Nov 03 00:32:38.171637 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:38.171639 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e3a97b9-b122-48e9-97c4-15b1a0e43e26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-wqw5q\" (UID: \"5e3a97b9-b122-48e9-97c4-15b1a0e43e26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wqw5q" Nov 03 00:32:38.171868 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:38.171700 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zslr4\" (UniqueName: \"kubernetes.io/projected/5e3a97b9-b122-48e9-97c4-15b1a0e43e26-kube-api-access-zslr4\") pod \"maas-api-revoke-sub-e2547622-wqw5q\" (UID: \"5e3a97b9-b122-48e9-97c4-15b1a0e43e26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wqw5q" Nov 03 00:32:38.172432 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:38.172408 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e3a97b9-b122-48e9-97c4-15b1a0e43e26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-wqw5q\" (UID: \"5e3a97b9-b122-48e9-97c4-15b1a0e43e26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wqw5q" Nov 03 00:32:38.174075 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:38.174050 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e3a97b9-b122-48e9-97c4-15b1a0e43e26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-wqw5q\" (UID: \"5e3a97b9-b122-48e9-97c4-15b1a0e43e26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wqw5q" Nov 03 00:32:38.226463 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:38.226430 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zslr4\" (UniqueName: \"kubernetes.io/projected/5e3a97b9-b122-48e9-97c4-15b1a0e43e26-kube-api-access-zslr4\") pod \"maas-api-revoke-sub-e2547622-wqw5q\" (UID: \"5e3a97b9-b122-48e9-97c4-15b1a0e43e26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wqw5q" Nov 03 00:32:38.304437 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:38.304357 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wqw5q" Nov 03 00:32:38.672624 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:38.672592 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wqw5q"] Nov 03 00:32:38.674038 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:32:38.673997 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e3a97b9_b122_48e9_97c4_15b1a0e43e26.slice/crio-04d197eb0ffe57c1b318af2e0431791a32c7e8793a35bc4c5b5cd9bb212b1505 WatchSource:0}: Error finding container 04d197eb0ffe57c1b318af2e0431791a32c7e8793a35bc4c5b5cd9bb212b1505: Status 404 returned error can't find the container with id 04d197eb0ffe57c1b318af2e0431791a32c7e8793a35bc4c5b5cd9bb212b1505 Nov 03 00:32:39.443273 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:39.443232 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wqw5q" event={"ID":"5e3a97b9-b122-48e9-97c4-15b1a0e43e26","Type":"ContainerStarted","Data":"fc4716fdc147f38ce68b3907ce4e1f2667cf9f8d9edb295c9bc45f1bb30d7aff"} Nov 03 00:32:39.443684 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:39.443290 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wqw5q" event={"ID":"5e3a97b9-b122-48e9-97c4-15b1a0e43e26","Type":"ContainerStarted","Data":"04d197eb0ffe57c1b318af2e0431791a32c7e8793a35bc4c5b5cd9bb212b1505"} Nov 03 00:32:39.554636 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:39.554584 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wqw5q" podStartSLOduration=2.554569674 podStartE2EDuration="2.554569674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:32:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:32:39.55277648 +0000 UTC m=+6008.521308265" watchObservedRunningTime="2026-11-03 00:32:39.554569674 +0000 UTC m=+6008.523101458" Nov 03 00:32:39.884042 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:32:39.883955 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:32:40.450746 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:40.450712 5076 generic.go:356] "Generic (PLEG): container finished" podID="5e3a97b9-b122-48e9-97c4-15b1a0e43e26" containerID="fc4716fdc147f38ce68b3907ce4e1f2667cf9f8d9edb295c9bc45f1bb30d7aff" exitCode=0 Nov 03 00:32:40.451274 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:40.450802 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wqw5q" event={"ID":"5e3a97b9-b122-48e9-97c4-15b1a0e43e26","Type":"ContainerDied","Data":"fc4716fdc147f38ce68b3907ce4e1f2667cf9f8d9edb295c9bc45f1bb30d7aff"} Nov 03 00:32:41.614864 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:41.614837 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wqw5q" Nov 03 00:32:41.707643 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:41.707556 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e3a97b9-b122-48e9-97c4-15b1a0e43e26-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e3a97b9-b122-48e9-97c4-15b1a0e43e26-maas-api-xks-ca\") pod \"5e3a97b9-b122-48e9-97c4-15b1a0e43e26\" (UID: \"5e3a97b9-b122-48e9-97c4-15b1a0e43e26\") " Nov 03 00:32:41.707643 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:41.707636 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e3a97b9-b122-48e9-97c4-15b1a0e43e26-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e3a97b9-b122-48e9-97c4-15b1a0e43e26-maas-api-openshift-ca\") pod \"5e3a97b9-b122-48e9-97c4-15b1a0e43e26\" (UID: \"5e3a97b9-b122-48e9-97c4-15b1a0e43e26\") " Nov 03 00:32:41.707879 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:41.707685 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e3a97b9-b122-48e9-97c4-15b1a0e43e26-kube-api-access-zslr4\" (UniqueName: \"kubernetes.io/projected/5e3a97b9-b122-48e9-97c4-15b1a0e43e26-kube-api-access-zslr4\") pod \"5e3a97b9-b122-48e9-97c4-15b1a0e43e26\" (UID: \"5e3a97b9-b122-48e9-97c4-15b1a0e43e26\") " Nov 03 00:32:41.708012 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:41.707986 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e3a97b9-b122-48e9-97c4-15b1a0e43e26-maas-api-openshift-ca" pod "5e3a97b9-b122-48e9-97c4-15b1a0e43e26" (UID: "5e3a97b9-b122-48e9-97c4-15b1a0e43e26"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:32:41.708130 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:41.708114 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e3a97b9-b122-48e9-97c4-15b1a0e43e26-maas-api-openshift-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:32:41.709757 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:41.709735 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e3a97b9-b122-48e9-97c4-15b1a0e43e26-maas-api-xks-ca" pod "5e3a97b9-b122-48e9-97c4-15b1a0e43e26" (UID: "5e3a97b9-b122-48e9-97c4-15b1a0e43e26"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:32:41.709757 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:41.709740 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e3a97b9-b122-48e9-97c4-15b1a0e43e26-kube-api-access-zslr4" pod "5e3a97b9-b122-48e9-97c4-15b1a0e43e26" (UID: "5e3a97b9-b122-48e9-97c4-15b1a0e43e26"). InnerVolumeSpecName "kube-api-access-zslr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:32:41.809032 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:41.808998 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e3a97b9-b122-48e9-97c4-15b1a0e43e26-maas-api-xks-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:32:41.809032 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:41.809028 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zslr4\" (UniqueName: \"kubernetes.io/projected/5e3a97b9-b122-48e9-97c4-15b1a0e43e26-kube-api-access-zslr4\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:32:42.462145 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:42.462111 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wqw5q" event={"ID":"5e3a97b9-b122-48e9-97c4-15b1a0e43e26","Type":"ContainerDied","Data":"04d197eb0ffe57c1b318af2e0431791a32c7e8793a35bc4c5b5cd9bb212b1505"} Nov 03 00:32:42.462332 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:42.462150 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wqw5q" Nov 03 00:32:42.462332 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:32:42.462162 5076 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04d197eb0ffe57c1b318af2e0431791a32c7e8793a35bc4c5b5cd9bb212b1505" Nov 03 00:33:09.624116 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:09.624081 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bbvrp"] Nov 03 00:33:09.624607 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:09.624536 5076 cpu_manager.go:414] "containerMap: removing container" podUID="5e3a97b9-b122-48e9-97c4-15b1a0e43e26" containerName="revoke-keys" Nov 03 00:33:09.624607 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:09.624549 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e3a97b9-b122-48e9-97c4-15b1a0e43e26" containerName="revoke-keys" Nov 03 00:33:09.624689 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:09.624621 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e3a97b9-b122-48e9-97c4-15b1a0e43e26" containerName="revoke-keys" Nov 03 00:33:09.628011 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:09.627995 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bbvrp" Nov 03 00:33:09.678101 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:09.678072 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:33:09.694810 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:09.694770 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5737a98-d253-46ba-a47a-73c2d4a84985-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-bbvrp\" (UID: \"e5737a98-d253-46ba-a47a-73c2d4a84985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bbvrp" Nov 03 00:33:09.695024 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:09.694824 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5737a98-d253-46ba-a47a-73c2d4a84985-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-bbvrp\" (UID: \"e5737a98-d253-46ba-a47a-73c2d4a84985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bbvrp" Nov 03 00:33:09.695024 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:09.694966 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftqvc\" (UniqueName: \"kubernetes.io/projected/e5737a98-d253-46ba-a47a-73c2d4a84985-kube-api-access-ftqvc\") pod \"maas-api-revoke-sub-c1e271a9-bbvrp\" (UID: \"e5737a98-d253-46ba-a47a-73c2d4a84985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bbvrp" Nov 03 00:33:09.725269 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:09.725238 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:33:09.727623 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:09.727601 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bbvrp"] Nov 03 00:33:09.795733 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:09.795702 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5737a98-d253-46ba-a47a-73c2d4a84985-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-bbvrp\" (UID: \"e5737a98-d253-46ba-a47a-73c2d4a84985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bbvrp" Nov 03 00:33:09.795910 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:09.795757 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5737a98-d253-46ba-a47a-73c2d4a84985-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-bbvrp\" (UID: \"e5737a98-d253-46ba-a47a-73c2d4a84985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bbvrp" Nov 03 00:33:09.795910 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:09.795821 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftqvc\" (UniqueName: \"kubernetes.io/projected/e5737a98-d253-46ba-a47a-73c2d4a84985-kube-api-access-ftqvc\") pod \"maas-api-revoke-sub-c1e271a9-bbvrp\" (UID: \"e5737a98-d253-46ba-a47a-73c2d4a84985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bbvrp" Nov 03 00:33:09.796509 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:09.796484 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5737a98-d253-46ba-a47a-73c2d4a84985-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-bbvrp\" (UID: \"e5737a98-d253-46ba-a47a-73c2d4a84985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bbvrp" Nov 03 00:33:09.798211 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:09.798188 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5737a98-d253-46ba-a47a-73c2d4a84985-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-bbvrp\" (UID: \"e5737a98-d253-46ba-a47a-73c2d4a84985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bbvrp" Nov 03 00:33:09.850052 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:09.850022 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftqvc\" (UniqueName: \"kubernetes.io/projected/e5737a98-d253-46ba-a47a-73c2d4a84985-kube-api-access-ftqvc\") pod \"maas-api-revoke-sub-c1e271a9-bbvrp\" (UID: \"e5737a98-d253-46ba-a47a-73c2d4a84985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bbvrp" Nov 03 00:33:09.884434 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:33:09.884349 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:33:09.941175 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:09.941141 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bbvrp" Nov 03 00:33:10.310402 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:10.310376 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bbvrp"] Nov 03 00:33:10.311425 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:33:10.311401 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5737a98_d253_46ba_a47a_73c2d4a84985.slice/crio-42a6024204bbbf87b850ee0207d0998d0d5108f75fd1d0575b4b5aaaabf83bc3 WatchSource:0}: Error finding container 42a6024204bbbf87b850ee0207d0998d0d5108f75fd1d0575b4b5aaaabf83bc3: Status 404 returned error can't find the container with id 42a6024204bbbf87b850ee0207d0998d0d5108f75fd1d0575b4b5aaaabf83bc3 Nov 03 00:33:10.650397 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:10.650369 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bbvrp" event={"ID":"e5737a98-d253-46ba-a47a-73c2d4a84985","Type":"ContainerStarted","Data":"a753bbf95f6932e0aa2e797169348957c0af0751c86a6e95aa9429b78e8d1ab6"} Nov 03 00:33:10.650751 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:10.650415 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bbvrp" event={"ID":"e5737a98-d253-46ba-a47a-73c2d4a84985","Type":"ContainerStarted","Data":"42a6024204bbbf87b850ee0207d0998d0d5108f75fd1d0575b4b5aaaabf83bc3"} Nov 03 00:33:10.761310 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:10.761258 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bbvrp" podStartSLOduration=1.761242813 podStartE2EDuration="1.761242813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:33:10.759182219 +0000 UTC m=+6039.727714004" watchObservedRunningTime="2026-11-03 00:33:10.761242813 +0000 UTC m=+6039.729774597" Nov 03 00:33:11.657476 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:11.657439 5076 generic.go:356] "Generic (PLEG): container finished" podID="e5737a98-d253-46ba-a47a-73c2d4a84985" containerID="a753bbf95f6932e0aa2e797169348957c0af0751c86a6e95aa9429b78e8d1ab6" exitCode=0 Nov 03 00:33:11.657959 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:11.657491 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bbvrp" event={"ID":"e5737a98-d253-46ba-a47a-73c2d4a84985","Type":"ContainerDied","Data":"a753bbf95f6932e0aa2e797169348957c0af0751c86a6e95aa9429b78e8d1ab6"} Nov 03 00:33:12.806338 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:12.806312 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bbvrp" Nov 03 00:33:12.830088 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:12.830061 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5737a98-d253-46ba-a47a-73c2d4a84985-kube-api-access-ftqvc\" (UniqueName: \"kubernetes.io/projected/e5737a98-d253-46ba-a47a-73c2d4a84985-kube-api-access-ftqvc\") pod \"e5737a98-d253-46ba-a47a-73c2d4a84985\" (UID: \"e5737a98-d253-46ba-a47a-73c2d4a84985\") " Nov 03 00:33:12.830215 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:12.830144 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5737a98-d253-46ba-a47a-73c2d4a84985-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5737a98-d253-46ba-a47a-73c2d4a84985-maas-api-xks-ca\") pod \"e5737a98-d253-46ba-a47a-73c2d4a84985\" (UID: \"e5737a98-d253-46ba-a47a-73c2d4a84985\") " Nov 03 00:33:12.830291 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:12.830275 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5737a98-d253-46ba-a47a-73c2d4a84985-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5737a98-d253-46ba-a47a-73c2d4a84985-maas-api-openshift-ca\") pod \"e5737a98-d253-46ba-a47a-73c2d4a84985\" (UID: \"e5737a98-d253-46ba-a47a-73c2d4a84985\") " Nov 03 00:33:12.830609 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:12.830586 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5737a98-d253-46ba-a47a-73c2d4a84985-maas-api-openshift-ca" pod "e5737a98-d253-46ba-a47a-73c2d4a84985" (UID: "e5737a98-d253-46ba-a47a-73c2d4a84985"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:33:12.832110 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:12.832082 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5737a98-d253-46ba-a47a-73c2d4a84985-kube-api-access-ftqvc" pod "e5737a98-d253-46ba-a47a-73c2d4a84985" (UID: "e5737a98-d253-46ba-a47a-73c2d4a84985"). InnerVolumeSpecName "kube-api-access-ftqvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:33:12.832110 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:12.832103 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5737a98-d253-46ba-a47a-73c2d4a84985-maas-api-xks-ca" pod "e5737a98-d253-46ba-a47a-73c2d4a84985" (UID: "e5737a98-d253-46ba-a47a-73c2d4a84985"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:33:12.931291 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:12.931211 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5737a98-d253-46ba-a47a-73c2d4a84985-maas-api-xks-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:33:12.931291 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:12.931239 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5737a98-d253-46ba-a47a-73c2d4a84985-maas-api-openshift-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:33:12.931291 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:12.931251 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftqvc\" (UniqueName: \"kubernetes.io/projected/e5737a98-d253-46ba-a47a-73c2d4a84985-kube-api-access-ftqvc\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:33:13.670144 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:13.670105 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bbvrp" event={"ID":"e5737a98-d253-46ba-a47a-73c2d4a84985","Type":"ContainerDied","Data":"42a6024204bbbf87b850ee0207d0998d0d5108f75fd1d0575b4b5aaaabf83bc3"} Nov 03 00:33:13.670310 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:13.670162 5076 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42a6024204bbbf87b850ee0207d0998d0d5108f75fd1d0575b4b5aaaabf83bc3" Nov 03 00:33:13.670310 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:13.670118 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bbvrp" Nov 03 00:33:34.360018 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:34.359981 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-25kwr"] Nov 03 00:33:34.360629 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:34.360464 5076 cpu_manager.go:414] "containerMap: removing container" podUID="e5737a98-d253-46ba-a47a-73c2d4a84985" containerName="revoke-keys" Nov 03 00:33:34.360629 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:34.360476 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5737a98-d253-46ba-a47a-73c2d4a84985" containerName="revoke-keys" Nov 03 00:33:34.360629 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:34.360553 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5737a98-d253-46ba-a47a-73c2d4a84985" containerName="revoke-keys" Nov 03 00:33:34.363786 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:34.363766 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-25kwr" Nov 03 00:33:34.415520 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:34.415480 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-25kwr"] Nov 03 00:33:34.436606 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:34.436565 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:33:34.460325 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:34.460294 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:33:34.488771 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:34.488739 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86vt4\" (UniqueName: \"kubernetes.io/projected/1dca5053-8077-4cd0-936d-6ff45551dac5-kube-api-access-86vt4\") pod \"maas-api-revoke-sub-2e1f5b5f-25kwr\" (UID: \"1dca5053-8077-4cd0-936d-6ff45551dac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-25kwr" Nov 03 00:33:34.488923 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:34.488871 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dca5053-8077-4cd0-936d-6ff45551dac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-25kwr\" (UID: \"1dca5053-8077-4cd0-936d-6ff45551dac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-25kwr" Nov 03 00:33:34.489010 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:34.488992 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dca5053-8077-4cd0-936d-6ff45551dac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-25kwr\" (UID: \"1dca5053-8077-4cd0-936d-6ff45551dac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-25kwr" Nov 03 00:33:34.590181 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:34.590141 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dca5053-8077-4cd0-936d-6ff45551dac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-25kwr\" (UID: \"1dca5053-8077-4cd0-936d-6ff45551dac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-25kwr" Nov 03 00:33:34.590365 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:34.590200 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dca5053-8077-4cd0-936d-6ff45551dac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-25kwr\" (UID: \"1dca5053-8077-4cd0-936d-6ff45551dac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-25kwr" Nov 03 00:33:34.590365 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:34.590281 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86vt4\" (UniqueName: \"kubernetes.io/projected/1dca5053-8077-4cd0-936d-6ff45551dac5-kube-api-access-86vt4\") pod \"maas-api-revoke-sub-2e1f5b5f-25kwr\" (UID: \"1dca5053-8077-4cd0-936d-6ff45551dac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-25kwr" Nov 03 00:33:34.590967 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:34.590939 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dca5053-8077-4cd0-936d-6ff45551dac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-25kwr\" (UID: \"1dca5053-8077-4cd0-936d-6ff45551dac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-25kwr" Nov 03 00:33:34.592619 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:34.592595 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dca5053-8077-4cd0-936d-6ff45551dac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-25kwr\" (UID: \"1dca5053-8077-4cd0-936d-6ff45551dac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-25kwr" Nov 03 00:33:34.644705 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:34.644629 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86vt4\" (UniqueName: \"kubernetes.io/projected/1dca5053-8077-4cd0-936d-6ff45551dac5-kube-api-access-86vt4\") pod \"maas-api-revoke-sub-2e1f5b5f-25kwr\" (UID: \"1dca5053-8077-4cd0-936d-6ff45551dac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-25kwr" Nov 03 00:33:34.677143 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:34.677112 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-25kwr" Nov 03 00:33:35.042756 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:35.042729 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-25kwr"] Nov 03 00:33:35.043982 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:33:35.043955 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dca5053_8077_4cd0_936d_6ff45551dac5.slice/crio-97737b4ed8c49c089142f7bf41c726fec3cd7c69e1f41b694f039d6834a36d1c WatchSource:0}: Error finding container 97737b4ed8c49c089142f7bf41c726fec3cd7c69e1f41b694f039d6834a36d1c: Status 404 returned error can't find the container with id 97737b4ed8c49c089142f7bf41c726fec3cd7c69e1f41b694f039d6834a36d1c Nov 03 00:33:35.810847 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:35.810811 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-25kwr" event={"ID":"1dca5053-8077-4cd0-936d-6ff45551dac5","Type":"ContainerStarted","Data":"eb8a468e570ea9e774305dcd3440d0aa922e8e6a9ed3d92abbcf5e550fb6baf3"} Nov 03 00:33:35.811242 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:35.810879 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-25kwr" event={"ID":"1dca5053-8077-4cd0-936d-6ff45551dac5","Type":"ContainerStarted","Data":"97737b4ed8c49c089142f7bf41c726fec3cd7c69e1f41b694f039d6834a36d1c"} Nov 03 00:33:35.931604 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:35.931549 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-25kwr" podStartSLOduration=1.931527615 podStartE2EDuration="1.931527615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:33:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:33:35.928166895 +0000 UTC m=+6064.896698669" watchObservedRunningTime="2026-11-03 00:33:35.931527615 +0000 UTC m=+6064.900059396" Nov 03 00:33:36.817758 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:36.817723 5076 generic.go:356] "Generic (PLEG): container finished" podID="1dca5053-8077-4cd0-936d-6ff45551dac5" containerID="eb8a468e570ea9e774305dcd3440d0aa922e8e6a9ed3d92abbcf5e550fb6baf3" exitCode=0 Nov 03 00:33:36.818176 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:36.817790 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-25kwr" event={"ID":"1dca5053-8077-4cd0-936d-6ff45551dac5","Type":"ContainerDied","Data":"eb8a468e570ea9e774305dcd3440d0aa922e8e6a9ed3d92abbcf5e550fb6baf3"} Nov 03 00:33:37.960621 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:37.960593 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-25kwr" Nov 03 00:33:38.125655 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:38.125572 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dca5053-8077-4cd0-936d-6ff45551dac5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dca5053-8077-4cd0-936d-6ff45551dac5-maas-api-xks-ca\") pod \"1dca5053-8077-4cd0-936d-6ff45551dac5\" (UID: \"1dca5053-8077-4cd0-936d-6ff45551dac5\") " Nov 03 00:33:38.125655 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:38.125623 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dca5053-8077-4cd0-936d-6ff45551dac5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dca5053-8077-4cd0-936d-6ff45551dac5-maas-api-openshift-ca\") pod \"1dca5053-8077-4cd0-936d-6ff45551dac5\" (UID: \"1dca5053-8077-4cd0-936d-6ff45551dac5\") " Nov 03 00:33:38.125852 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:38.125694 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dca5053-8077-4cd0-936d-6ff45551dac5-kube-api-access-86vt4\" (UniqueName: \"kubernetes.io/projected/1dca5053-8077-4cd0-936d-6ff45551dac5-kube-api-access-86vt4\") pod \"1dca5053-8077-4cd0-936d-6ff45551dac5\" (UID: \"1dca5053-8077-4cd0-936d-6ff45551dac5\") " Nov 03 00:33:38.126069 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:38.126047 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dca5053-8077-4cd0-936d-6ff45551dac5-maas-api-openshift-ca" pod "1dca5053-8077-4cd0-936d-6ff45551dac5" (UID: "1dca5053-8077-4cd0-936d-6ff45551dac5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:33:38.127885 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:38.127857 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dca5053-8077-4cd0-936d-6ff45551dac5-maas-api-xks-ca" pod "1dca5053-8077-4cd0-936d-6ff45551dac5" (UID: "1dca5053-8077-4cd0-936d-6ff45551dac5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:33:38.127885 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:38.127866 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dca5053-8077-4cd0-936d-6ff45551dac5-kube-api-access-86vt4" pod "1dca5053-8077-4cd0-936d-6ff45551dac5" (UID: "1dca5053-8077-4cd0-936d-6ff45551dac5"). InnerVolumeSpecName "kube-api-access-86vt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:33:38.227325 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:38.227288 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86vt4\" (UniqueName: \"kubernetes.io/projected/1dca5053-8077-4cd0-936d-6ff45551dac5-kube-api-access-86vt4\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:33:38.227325 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:38.227321 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dca5053-8077-4cd0-936d-6ff45551dac5-maas-api-xks-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:33:38.227325 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:38.227333 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dca5053-8077-4cd0-936d-6ff45551dac5-maas-api-openshift-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:33:38.833511 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:38.833471 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-25kwr" event={"ID":"1dca5053-8077-4cd0-936d-6ff45551dac5","Type":"ContainerDied","Data":"97737b4ed8c49c089142f7bf41c726fec3cd7c69e1f41b694f039d6834a36d1c"} Nov 03 00:33:38.833834 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:38.833803 5076 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97737b4ed8c49c089142f7bf41c726fec3cd7c69e1f41b694f039d6834a36d1c" Nov 03 00:33:38.833834 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:38.833633 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-25kwr" Nov 03 00:33:39.883180 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:33:39.883144 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:33:46.990643 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:46.990557 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8tbdm"] Nov 03 00:33:46.991287 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:46.991266 5076 cpu_manager.go:414] "containerMap: removing container" podUID="1dca5053-8077-4cd0-936d-6ff45551dac5" containerName="revoke-keys" Nov 03 00:33:46.991333 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:46.991290 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dca5053-8077-4cd0-936d-6ff45551dac5" containerName="revoke-keys" Nov 03 00:33:46.991411 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:46.991400 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dca5053-8077-4cd0-936d-6ff45551dac5" containerName="revoke-keys" Nov 03 00:33:46.995283 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:46.995263 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8tbdm" Nov 03 00:33:47.009193 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:47.009162 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45lw7\" (UniqueName: \"kubernetes.io/projected/3202fb7b-a304-4b46-8663-57d35427b1ca-kube-api-access-45lw7\") pod \"maas-api-revoke-sub-ec843018-8tbdm\" (UID: \"3202fb7b-a304-4b46-8663-57d35427b1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8tbdm" Nov 03 00:33:47.009334 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:47.009246 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3202fb7b-a304-4b46-8663-57d35427b1ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-8tbdm\" (UID: \"3202fb7b-a304-4b46-8663-57d35427b1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8tbdm" Nov 03 00:33:47.009334 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:47.009293 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3202fb7b-a304-4b46-8663-57d35427b1ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-8tbdm\" (UID: \"3202fb7b-a304-4b46-8663-57d35427b1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8tbdm" Nov 03 00:33:47.047380 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:47.047348 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8tbdm"] Nov 03 00:33:47.092102 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:47.092071 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:33:47.092102 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:47.092094 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:33:47.109945 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:47.109886 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3202fb7b-a304-4b46-8663-57d35427b1ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-8tbdm\" (UID: \"3202fb7b-a304-4b46-8663-57d35427b1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8tbdm" Nov 03 00:33:47.110146 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:47.109977 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3202fb7b-a304-4b46-8663-57d35427b1ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-8tbdm\" (UID: \"3202fb7b-a304-4b46-8663-57d35427b1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8tbdm" Nov 03 00:33:47.110146 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:47.110073 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45lw7\" (UniqueName: \"kubernetes.io/projected/3202fb7b-a304-4b46-8663-57d35427b1ca-kube-api-access-45lw7\") pod \"maas-api-revoke-sub-ec843018-8tbdm\" (UID: \"3202fb7b-a304-4b46-8663-57d35427b1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8tbdm" Nov 03 00:33:47.110704 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:47.110684 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3202fb7b-a304-4b46-8663-57d35427b1ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-8tbdm\" (UID: \"3202fb7b-a304-4b46-8663-57d35427b1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8tbdm" Nov 03 00:33:47.112171 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:47.112152 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3202fb7b-a304-4b46-8663-57d35427b1ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-8tbdm\" (UID: \"3202fb7b-a304-4b46-8663-57d35427b1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8tbdm" Nov 03 00:33:47.165716 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:47.165681 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45lw7\" (UniqueName: \"kubernetes.io/projected/3202fb7b-a304-4b46-8663-57d35427b1ca-kube-api-access-45lw7\") pod \"maas-api-revoke-sub-ec843018-8tbdm\" (UID: \"3202fb7b-a304-4b46-8663-57d35427b1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8tbdm" Nov 03 00:33:47.309314 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:47.309233 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8tbdm" Nov 03 00:33:47.676799 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:47.676770 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8tbdm"] Nov 03 00:33:47.677988 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:33:47.677958 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3202fb7b_a304_4b46_8663_57d35427b1ca.slice/crio-ceac5bc61a20301feebf10c4989fea924593af7254f6fdfd887fdbfa5fe27e6e WatchSource:0}: Error finding container ceac5bc61a20301feebf10c4989fea924593af7254f6fdfd887fdbfa5fe27e6e: Status 404 returned error can't find the container with id ceac5bc61a20301feebf10c4989fea924593af7254f6fdfd887fdbfa5fe27e6e Nov 03 00:33:47.885284 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:47.885251 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8tbdm" event={"ID":"3202fb7b-a304-4b46-8663-57d35427b1ca","Type":"ContainerStarted","Data":"ceac5bc61a20301feebf10c4989fea924593af7254f6fdfd887fdbfa5fe27e6e"} Nov 03 00:33:48.892855 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:48.892770 5076 generic.go:356] "Generic (PLEG): container finished" podID="3202fb7b-a304-4b46-8663-57d35427b1ca" containerID="69a72a6e60b45e6b84b8731c66d60c99ce7d475bfd8610dae4daf4f479d6f95f" exitCode=0 Nov 03 00:33:48.893249 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:48.892848 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8tbdm" event={"ID":"3202fb7b-a304-4b46-8663-57d35427b1ca","Type":"ContainerDied","Data":"69a72a6e60b45e6b84b8731c66d60c99ce7d475bfd8610dae4daf4f479d6f95f"} Nov 03 00:33:50.061600 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:50.061575 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8tbdm" Nov 03 00:33:50.141735 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:50.141698 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3202fb7b-a304-4b46-8663-57d35427b1ca-kube-api-access-45lw7\" (UniqueName: \"kubernetes.io/projected/3202fb7b-a304-4b46-8663-57d35427b1ca-kube-api-access-45lw7\") pod \"3202fb7b-a304-4b46-8663-57d35427b1ca\" (UID: \"3202fb7b-a304-4b46-8663-57d35427b1ca\") " Nov 03 00:33:50.141932 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:50.141797 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3202fb7b-a304-4b46-8663-57d35427b1ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3202fb7b-a304-4b46-8663-57d35427b1ca-maas-api-openshift-ca\") pod \"3202fb7b-a304-4b46-8663-57d35427b1ca\" (UID: \"3202fb7b-a304-4b46-8663-57d35427b1ca\") " Nov 03 00:33:50.141932 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:50.141866 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3202fb7b-a304-4b46-8663-57d35427b1ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3202fb7b-a304-4b46-8663-57d35427b1ca-maas-api-xks-ca\") pod \"3202fb7b-a304-4b46-8663-57d35427b1ca\" (UID: \"3202fb7b-a304-4b46-8663-57d35427b1ca\") " Nov 03 00:33:50.142234 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:50.142210 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3202fb7b-a304-4b46-8663-57d35427b1ca-maas-api-openshift-ca" pod "3202fb7b-a304-4b46-8663-57d35427b1ca" (UID: "3202fb7b-a304-4b46-8663-57d35427b1ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:33:50.143927 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:50.143876 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3202fb7b-a304-4b46-8663-57d35427b1ca-maas-api-xks-ca" pod "3202fb7b-a304-4b46-8663-57d35427b1ca" (UID: "3202fb7b-a304-4b46-8663-57d35427b1ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:33:50.144026 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:50.143959 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3202fb7b-a304-4b46-8663-57d35427b1ca-kube-api-access-45lw7" pod "3202fb7b-a304-4b46-8663-57d35427b1ca" (UID: "3202fb7b-a304-4b46-8663-57d35427b1ca"). InnerVolumeSpecName "kube-api-access-45lw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:33:50.243205 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:50.243166 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3202fb7b-a304-4b46-8663-57d35427b1ca-maas-api-xks-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:33:50.243205 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:50.243199 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45lw7\" (UniqueName: \"kubernetes.io/projected/3202fb7b-a304-4b46-8663-57d35427b1ca-kube-api-access-45lw7\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:33:50.243205 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:50.243211 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3202fb7b-a304-4b46-8663-57d35427b1ca-maas-api-openshift-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:33:50.905376 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:50.905346 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8tbdm" Nov 03 00:33:50.905555 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:50.905351 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8tbdm" event={"ID":"3202fb7b-a304-4b46-8663-57d35427b1ca","Type":"ContainerDied","Data":"ceac5bc61a20301feebf10c4989fea924593af7254f6fdfd887fdbfa5fe27e6e"} Nov 03 00:33:50.905555 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:33:50.905469 5076 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ceac5bc61a20301feebf10c4989fea924593af7254f6fdfd887fdbfa5fe27e6e" Nov 03 00:34:09.883849 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:34:09.883808 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:34:13.046297 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:13.046260 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2wx9v"] Nov 03 00:34:13.092705 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:13.092672 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2wx9v"] Nov 03 00:34:13.149047 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:13.149008 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75e5e93a-6f2e-48bc-9e90-21fd36262cd0" path="/var/lib/kubelet/pods/75e5e93a-6f2e-48bc-9e90-21fd36262cd0/volumes" Nov 03 00:34:22.086354 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:22.086320 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xbb6j"] Nov 03 00:34:22.086818 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:22.086803 5076 cpu_manager.go:414] "containerMap: removing container" podUID="3202fb7b-a304-4b46-8663-57d35427b1ca" containerName="revoke-keys" Nov 03 00:34:22.086865 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:22.086819 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3202fb7b-a304-4b46-8663-57d35427b1ca" containerName="revoke-keys" Nov 03 00:34:22.086951 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:22.086911 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="3202fb7b-a304-4b46-8663-57d35427b1ca" containerName="revoke-keys" Nov 03 00:34:22.090010 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:22.089993 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xbb6j" Nov 03 00:34:22.140672 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:22.140626 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:34:22.188382 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:22.188345 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:34:22.191026 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:22.190996 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xbb6j"] Nov 03 00:34:22.194144 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:22.194121 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/399e376c-5a9e-4d79-9f04-6c5bb148acbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-xbb6j\" (UID: \"399e376c-5a9e-4d79-9f04-6c5bb148acbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xbb6j" Nov 03 00:34:22.194261 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:22.194161 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9q6s\" (UniqueName: \"kubernetes.io/projected/399e376c-5a9e-4d79-9f04-6c5bb148acbf-kube-api-access-q9q6s\") pod \"maas-api-revoke-sub-046e1343-xbb6j\" (UID: \"399e376c-5a9e-4d79-9f04-6c5bb148acbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xbb6j" Nov 03 00:34:22.194261 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:22.194191 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/399e376c-5a9e-4d79-9f04-6c5bb148acbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-xbb6j\" (UID: \"399e376c-5a9e-4d79-9f04-6c5bb148acbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xbb6j" Nov 03 00:34:22.295505 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:22.295469 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/399e376c-5a9e-4d79-9f04-6c5bb148acbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-xbb6j\" (UID: \"399e376c-5a9e-4d79-9f04-6c5bb148acbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xbb6j" Nov 03 00:34:22.295717 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:22.295521 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9q6s\" (UniqueName: \"kubernetes.io/projected/399e376c-5a9e-4d79-9f04-6c5bb148acbf-kube-api-access-q9q6s\") pod \"maas-api-revoke-sub-046e1343-xbb6j\" (UID: \"399e376c-5a9e-4d79-9f04-6c5bb148acbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xbb6j" Nov 03 00:34:22.295717 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:22.295571 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/399e376c-5a9e-4d79-9f04-6c5bb148acbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-xbb6j\" (UID: \"399e376c-5a9e-4d79-9f04-6c5bb148acbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xbb6j" Nov 03 00:34:22.296194 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:22.296164 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/399e376c-5a9e-4d79-9f04-6c5bb148acbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-xbb6j\" (UID: \"399e376c-5a9e-4d79-9f04-6c5bb148acbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xbb6j" Nov 03 00:34:22.297928 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:22.297908 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/399e376c-5a9e-4d79-9f04-6c5bb148acbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-xbb6j\" (UID: \"399e376c-5a9e-4d79-9f04-6c5bb148acbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xbb6j" Nov 03 00:34:22.353199 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:22.353123 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9q6s\" (UniqueName: \"kubernetes.io/projected/399e376c-5a9e-4d79-9f04-6c5bb148acbf-kube-api-access-q9q6s\") pod \"maas-api-revoke-sub-046e1343-xbb6j\" (UID: \"399e376c-5a9e-4d79-9f04-6c5bb148acbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xbb6j" Nov 03 00:34:22.402970 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:22.402939 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xbb6j" Nov 03 00:34:22.770407 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:22.770381 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xbb6j"] Nov 03 00:34:22.771856 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:34:22.771821 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod399e376c_5a9e_4d79_9f04_6c5bb148acbf.slice/crio-b4be4cdcce15b860e357a26b29f0cc7f23dd6a432f98fa0859c3776788e03045 WatchSource:0}: Error finding container b4be4cdcce15b860e357a26b29f0cc7f23dd6a432f98fa0859c3776788e03045: Status 404 returned error can't find the container with id b4be4cdcce15b860e357a26b29f0cc7f23dd6a432f98fa0859c3776788e03045 Nov 03 00:34:23.088130 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:23.088051 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xbb6j" event={"ID":"399e376c-5a9e-4d79-9f04-6c5bb148acbf","Type":"ContainerStarted","Data":"b4be4cdcce15b860e357a26b29f0cc7f23dd6a432f98fa0859c3776788e03045"} Nov 03 00:34:24.094819 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:24.094785 5076 generic.go:356] "Generic (PLEG): container finished" podID="399e376c-5a9e-4d79-9f04-6c5bb148acbf" containerID="221685e77f34d3b46086d18eaf489569689b877bac3e34e32b262096631aa284" exitCode=0 Nov 03 00:34:24.095242 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:24.094843 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xbb6j" event={"ID":"399e376c-5a9e-4d79-9f04-6c5bb148acbf","Type":"ContainerDied","Data":"221685e77f34d3b46086d18eaf489569689b877bac3e34e32b262096631aa284"} Nov 03 00:34:25.245428 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:25.245399 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xbb6j" Nov 03 00:34:25.326881 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:25.326845 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/399e376c-5a9e-4d79-9f04-6c5bb148acbf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/399e376c-5a9e-4d79-9f04-6c5bb148acbf-maas-api-openshift-ca\") pod \"399e376c-5a9e-4d79-9f04-6c5bb148acbf\" (UID: \"399e376c-5a9e-4d79-9f04-6c5bb148acbf\") " Nov 03 00:34:25.327075 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:25.326969 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/399e376c-5a9e-4d79-9f04-6c5bb148acbf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/399e376c-5a9e-4d79-9f04-6c5bb148acbf-maas-api-xks-ca\") pod \"399e376c-5a9e-4d79-9f04-6c5bb148acbf\" (UID: \"399e376c-5a9e-4d79-9f04-6c5bb148acbf\") " Nov 03 00:34:25.327075 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:25.327045 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/399e376c-5a9e-4d79-9f04-6c5bb148acbf-kube-api-access-q9q6s\" (UniqueName: \"kubernetes.io/projected/399e376c-5a9e-4d79-9f04-6c5bb148acbf-kube-api-access-q9q6s\") pod \"399e376c-5a9e-4d79-9f04-6c5bb148acbf\" (UID: \"399e376c-5a9e-4d79-9f04-6c5bb148acbf\") " Nov 03 00:34:25.327277 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:25.327255 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/399e376c-5a9e-4d79-9f04-6c5bb148acbf-maas-api-openshift-ca" pod "399e376c-5a9e-4d79-9f04-6c5bb148acbf" (UID: "399e376c-5a9e-4d79-9f04-6c5bb148acbf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:34:25.327410 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:25.327394 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/399e376c-5a9e-4d79-9f04-6c5bb148acbf-maas-api-openshift-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:34:25.329027 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:25.329007 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/399e376c-5a9e-4d79-9f04-6c5bb148acbf-maas-api-xks-ca" pod "399e376c-5a9e-4d79-9f04-6c5bb148acbf" (UID: "399e376c-5a9e-4d79-9f04-6c5bb148acbf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:34:25.329115 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:25.329028 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/399e376c-5a9e-4d79-9f04-6c5bb148acbf-kube-api-access-q9q6s" pod "399e376c-5a9e-4d79-9f04-6c5bb148acbf" (UID: "399e376c-5a9e-4d79-9f04-6c5bb148acbf"). InnerVolumeSpecName "kube-api-access-q9q6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:34:25.428213 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:25.428108 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9q6s\" (UniqueName: \"kubernetes.io/projected/399e376c-5a9e-4d79-9f04-6c5bb148acbf-kube-api-access-q9q6s\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:34:25.428213 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:25.428146 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/399e376c-5a9e-4d79-9f04-6c5bb148acbf-maas-api-xks-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:34:26.106353 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:26.106321 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xbb6j" Nov 03 00:34:26.106353 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:26.106337 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xbb6j" event={"ID":"399e376c-5a9e-4d79-9f04-6c5bb148acbf","Type":"ContainerDied","Data":"b4be4cdcce15b860e357a26b29f0cc7f23dd6a432f98fa0859c3776788e03045"} Nov 03 00:34:26.106668 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:26.106410 5076 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4be4cdcce15b860e357a26b29f0cc7f23dd6a432f98fa0859c3776788e03045" Nov 03 00:34:31.045357 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:31.045322 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zhks7"] Nov 03 00:34:31.094113 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:31.094074 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-zhks7"] Nov 03 00:34:31.147695 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:31.147661 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c183247-b2f3-4f68-afa2-233a8a8bb647" path="/var/lib/kubelet/pods/0c183247-b2f3-4f68-afa2-233a8a8bb647/volumes" Nov 03 00:34:32.904126 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:32.904089 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-67zfm"] Nov 03 00:34:32.904558 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:32.904545 5076 cpu_manager.go:414] "containerMap: removing container" podUID="399e376c-5a9e-4d79-9f04-6c5bb148acbf" containerName="revoke-keys" Nov 03 00:34:32.904610 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:32.904559 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="399e376c-5a9e-4d79-9f04-6c5bb148acbf" containerName="revoke-keys" Nov 03 00:34:32.904649 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:32.904634 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="399e376c-5a9e-4d79-9f04-6c5bb148acbf" containerName="revoke-keys" Nov 03 00:34:32.908129 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:32.908112 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-67zfm" Nov 03 00:34:32.909679 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:32.909655 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6jqv\" (UniqueName: \"kubernetes.io/projected/36b9297b-e093-4d79-980c-a208a4d51154-kube-api-access-d6jqv\") pod \"maas-api-revoke-sub-d3fba056-67zfm\" (UID: \"36b9297b-e093-4d79-980c-a208a4d51154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-67zfm" Nov 03 00:34:32.909761 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:32.909709 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36b9297b-e093-4d79-980c-a208a4d51154-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-67zfm\" (UID: \"36b9297b-e093-4d79-980c-a208a4d51154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-67zfm" Nov 03 00:34:32.909805 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:32.909789 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36b9297b-e093-4d79-980c-a208a4d51154-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-67zfm\" (UID: \"36b9297b-e093-4d79-980c-a208a4d51154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-67zfm" Nov 03 00:34:33.005776 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:33.005744 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:34:33.005983 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:33.005798 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:34:33.008251 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:33.008228 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-67zfm"] Nov 03 00:34:33.010385 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:33.010365 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6jqv\" (UniqueName: \"kubernetes.io/projected/36b9297b-e093-4d79-980c-a208a4d51154-kube-api-access-d6jqv\") pod \"maas-api-revoke-sub-d3fba056-67zfm\" (UID: \"36b9297b-e093-4d79-980c-a208a4d51154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-67zfm" Nov 03 00:34:33.010442 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:33.010415 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36b9297b-e093-4d79-980c-a208a4d51154-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-67zfm\" (UID: \"36b9297b-e093-4d79-980c-a208a4d51154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-67zfm" Nov 03 00:34:33.010484 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:33.010442 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36b9297b-e093-4d79-980c-a208a4d51154-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-67zfm\" (UID: \"36b9297b-e093-4d79-980c-a208a4d51154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-67zfm" Nov 03 00:34:33.011086 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:33.011067 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36b9297b-e093-4d79-980c-a208a4d51154-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-67zfm\" (UID: \"36b9297b-e093-4d79-980c-a208a4d51154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-67zfm" Nov 03 00:34:33.012799 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:33.012782 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36b9297b-e093-4d79-980c-a208a4d51154-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-67zfm\" (UID: \"36b9297b-e093-4d79-980c-a208a4d51154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-67zfm" Nov 03 00:34:33.067844 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:33.067816 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6jqv\" (UniqueName: \"kubernetes.io/projected/36b9297b-e093-4d79-980c-a208a4d51154-kube-api-access-d6jqv\") pod \"maas-api-revoke-sub-d3fba056-67zfm\" (UID: \"36b9297b-e093-4d79-980c-a208a4d51154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-67zfm" Nov 03 00:34:33.220678 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:33.220648 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-67zfm" Nov 03 00:34:33.585714 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:33.585684 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-67zfm"] Nov 03 00:34:33.586697 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:34:33.586665 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36b9297b_e093_4d79_980c_a208a4d51154.slice/crio-cb653ed557cc52fc0f5e42817632e39b249dc530bdfbc1869d47ddced61a968e WatchSource:0}: Error finding container cb653ed557cc52fc0f5e42817632e39b249dc530bdfbc1869d47ddced61a968e: Status 404 returned error can't find the container with id cb653ed557cc52fc0f5e42817632e39b249dc530bdfbc1869d47ddced61a968e Nov 03 00:34:34.152710 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:34.152674 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-67zfm" event={"ID":"36b9297b-e093-4d79-980c-a208a4d51154","Type":"ContainerStarted","Data":"4bcd429f27383747ba395f569b0784c72192d7419a97e60711336dd7d45f0d83"} Nov 03 00:34:34.153126 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:34.152732 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-67zfm" event={"ID":"36b9297b-e093-4d79-980c-a208a4d51154","Type":"ContainerStarted","Data":"cb653ed557cc52fc0f5e42817632e39b249dc530bdfbc1869d47ddced61a968e"} Nov 03 00:34:34.266453 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:34.266406 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-67zfm" podStartSLOduration=2.266392839 podStartE2EDuration="2.266392839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:34:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:34:34.265833566 +0000 UTC m=+6123.234365350" watchObservedRunningTime="2026-11-03 00:34:34.266392839 +0000 UTC m=+6123.234924667" Nov 03 00:34:35.158486 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:35.158447 5076 generic.go:356] "Generic (PLEG): container finished" podID="36b9297b-e093-4d79-980c-a208a4d51154" containerID="4bcd429f27383747ba395f569b0784c72192d7419a97e60711336dd7d45f0d83" exitCode=0 Nov 03 00:34:35.158904 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:35.158528 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-67zfm" event={"ID":"36b9297b-e093-4d79-980c-a208a4d51154","Type":"ContainerDied","Data":"4bcd429f27383747ba395f569b0784c72192d7419a97e60711336dd7d45f0d83"} Nov 03 00:34:36.370283 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:36.370250 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-67zfm" Nov 03 00:34:36.443840 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:36.443791 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36b9297b-e093-4d79-980c-a208a4d51154-kube-api-access-d6jqv\" (UniqueName: \"kubernetes.io/projected/36b9297b-e093-4d79-980c-a208a4d51154-kube-api-access-d6jqv\") pod \"36b9297b-e093-4d79-980c-a208a4d51154\" (UID: \"36b9297b-e093-4d79-980c-a208a4d51154\") " Nov 03 00:34:36.444081 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:36.443927 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36b9297b-e093-4d79-980c-a208a4d51154-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36b9297b-e093-4d79-980c-a208a4d51154-maas-api-openshift-ca\") pod \"36b9297b-e093-4d79-980c-a208a4d51154\" (UID: \"36b9297b-e093-4d79-980c-a208a4d51154\") " Nov 03 00:34:36.444081 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:36.444008 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36b9297b-e093-4d79-980c-a208a4d51154-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36b9297b-e093-4d79-980c-a208a4d51154-maas-api-xks-ca\") pod \"36b9297b-e093-4d79-980c-a208a4d51154\" (UID: \"36b9297b-e093-4d79-980c-a208a4d51154\") " Nov 03 00:34:36.444614 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:36.444580 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36b9297b-e093-4d79-980c-a208a4d51154-maas-api-openshift-ca" pod "36b9297b-e093-4d79-980c-a208a4d51154" (UID: "36b9297b-e093-4d79-980c-a208a4d51154"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:34:36.446746 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:36.446712 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b9297b-e093-4d79-980c-a208a4d51154-kube-api-access-d6jqv" pod "36b9297b-e093-4d79-980c-a208a4d51154" (UID: "36b9297b-e093-4d79-980c-a208a4d51154"). InnerVolumeSpecName "kube-api-access-d6jqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:34:36.446864 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:36.446824 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36b9297b-e093-4d79-980c-a208a4d51154-maas-api-xks-ca" pod "36b9297b-e093-4d79-980c-a208a4d51154" (UID: "36b9297b-e093-4d79-980c-a208a4d51154"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:34:36.568917 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:36.565553 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6jqv\" (UniqueName: \"kubernetes.io/projected/36b9297b-e093-4d79-980c-a208a4d51154-kube-api-access-d6jqv\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:34:36.568917 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:36.565596 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36b9297b-e093-4d79-980c-a208a4d51154-maas-api-openshift-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:34:36.568917 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:36.565614 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36b9297b-e093-4d79-980c-a208a4d51154-maas-api-xks-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:34:37.220931 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:37.220831 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-67zfm" event={"ID":"36b9297b-e093-4d79-980c-a208a4d51154","Type":"ContainerDied","Data":"cb653ed557cc52fc0f5e42817632e39b249dc530bdfbc1869d47ddced61a968e"} Nov 03 00:34:37.227615 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:37.221168 5076 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb653ed557cc52fc0f5e42817632e39b249dc530bdfbc1869d47ddced61a968e" Nov 03 00:34:37.227615 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:37.221274 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-67zfm" Nov 03 00:34:39.884204 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:34:39.884164 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:34:55.044406 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:55.044366 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83a4f24f-4ppm6"] Nov 03 00:34:55.093256 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:55.093219 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83a4f24f-4ppm6"] Nov 03 00:34:55.146983 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:34:55.146950 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79069e48-269e-42bc-a59a-a71e9c1aea4a" path="/var/lib/kubelet/pods/79069e48-269e-42bc-a59a-a71e9c1aea4a/volumes" Nov 03 00:35:07.781627 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:07.781596 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d84vk"] Nov 03 00:35:07.782112 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:07.782055 5076 cpu_manager.go:414] "containerMap: removing container" podUID="36b9297b-e093-4d79-980c-a208a4d51154" containerName="revoke-keys" Nov 03 00:35:07.782112 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:07.782067 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36b9297b-e093-4d79-980c-a208a4d51154" containerName="revoke-keys" Nov 03 00:35:07.782190 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:07.782170 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="36b9297b-e093-4d79-980c-a208a4d51154" containerName="revoke-keys" Nov 03 00:35:07.785711 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:07.785690 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d84vk" Nov 03 00:35:07.883016 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:07.882981 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:35:07.883199 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:07.883031 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:35:07.885471 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:07.885447 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d84vk"] Nov 03 00:35:07.952945 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:07.952880 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-d84vk\" (UID: \"ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d84vk" Nov 03 00:35:07.953126 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:07.952956 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s68bw\" (UniqueName: \"kubernetes.io/projected/ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5-kube-api-access-s68bw\") pod \"maas-api-revoke-sub-ae594979-d84vk\" (UID: \"ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d84vk" Nov 03 00:35:07.953126 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:07.953025 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-d84vk\" (UID: \"ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d84vk" Nov 03 00:35:08.054474 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:08.054387 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-d84vk\" (UID: \"ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d84vk" Nov 03 00:35:08.054474 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:08.054430 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s68bw\" (UniqueName: \"kubernetes.io/projected/ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5-kube-api-access-s68bw\") pod \"maas-api-revoke-sub-ae594979-d84vk\" (UID: \"ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d84vk" Nov 03 00:35:08.054474 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:08.054473 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-d84vk\" (UID: \"ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d84vk" Nov 03 00:35:08.055121 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:08.055097 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-d84vk\" (UID: \"ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d84vk" Nov 03 00:35:08.056869 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:08.056847 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-d84vk\" (UID: \"ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d84vk" Nov 03 00:35:08.108836 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:08.108806 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s68bw\" (UniqueName: \"kubernetes.io/projected/ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5-kube-api-access-s68bw\") pod \"maas-api-revoke-sub-ae594979-d84vk\" (UID: \"ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d84vk" Nov 03 00:35:08.398926 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:08.398818 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d84vk" Nov 03 00:35:08.767094 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:08.767064 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d84vk"] Nov 03 00:35:08.769118 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:35:08.769089 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee4e4b4b_c5c8_4d09_9c02_da5bc3cdfcd5.slice/crio-c42365fc82ec880a1b518fde78b42009cb7797f15492473c35f412f93bc2ee92 WatchSource:0}: Error finding container c42365fc82ec880a1b518fde78b42009cb7797f15492473c35f412f93bc2ee92: Status 404 returned error can't find the container with id c42365fc82ec880a1b518fde78b42009cb7797f15492473c35f412f93bc2ee92 Nov 03 00:35:09.458877 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:09.458842 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d84vk" event={"ID":"ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5","Type":"ContainerStarted","Data":"6c9c6c51cd53462ad56d06d481de3b01c1c5a8b080b1f93e1b5c4be8ef0c201c"} Nov 03 00:35:09.459400 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:09.458922 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d84vk" event={"ID":"ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5","Type":"ContainerStarted","Data":"c42365fc82ec880a1b518fde78b42009cb7797f15492473c35f412f93bc2ee92"} Nov 03 00:35:09.568792 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:09.568735 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d84vk" podStartSLOduration=2.568720878 podStartE2EDuration="2.568720878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:09.56735237 +0000 UTC m=+6158.535884154" watchObservedRunningTime="2026-11-03 00:35:09.568720878 +0000 UTC m=+6158.537252656" Nov 03 00:35:09.883987 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:35:09.883898 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:35:10.465398 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:10.465367 5076 generic.go:356] "Generic (PLEG): container finished" podID="ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5" containerID="6c9c6c51cd53462ad56d06d481de3b01c1c5a8b080b1f93e1b5c4be8ef0c201c" exitCode=0 Nov 03 00:35:10.465800 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:10.465427 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d84vk" event={"ID":"ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5","Type":"ContainerDied","Data":"6c9c6c51cd53462ad56d06d481de3b01c1c5a8b080b1f93e1b5c4be8ef0c201c"} Nov 03 00:35:11.613179 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:11.613152 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d84vk" Nov 03 00:35:11.790296 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:11.790208 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5-maas-api-xks-ca\") pod \"ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5\" (UID: \"ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5\") " Nov 03 00:35:11.790296 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:11.790279 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5-maas-api-openshift-ca\") pod \"ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5\" (UID: \"ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5\") " Nov 03 00:35:11.790495 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:11.790364 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5-kube-api-access-s68bw\" (UniqueName: \"kubernetes.io/projected/ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5-kube-api-access-s68bw\") pod \"ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5\" (UID: \"ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5\") " Nov 03 00:35:11.790738 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:11.790707 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5-maas-api-openshift-ca" pod "ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5" (UID: "ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:35:11.792454 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:11.792428 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5-maas-api-xks-ca" pod "ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5" (UID: "ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:35:11.792530 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:11.792473 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5-kube-api-access-s68bw" pod "ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5" (UID: "ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5"). InnerVolumeSpecName "kube-api-access-s68bw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:35:11.891857 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:11.891812 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5-maas-api-xks-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:35:11.891857 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:11.891853 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5-maas-api-openshift-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:35:11.891857 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:11.891868 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s68bw\" (UniqueName: \"kubernetes.io/projected/ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5-kube-api-access-s68bw\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:35:12.477226 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:12.477198 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d84vk" Nov 03 00:35:12.477402 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:12.477203 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d84vk" event={"ID":"ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5","Type":"ContainerDied","Data":"c42365fc82ec880a1b518fde78b42009cb7797f15492473c35f412f93bc2ee92"} Nov 03 00:35:12.477402 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:12.477325 5076 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c42365fc82ec880a1b518fde78b42009cb7797f15492473c35f412f93bc2ee92" Nov 03 00:35:29.947487 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:29.947449 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9zmbh"] Nov 03 00:35:29.948211 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:29.948190 5076 cpu_manager.go:414] "containerMap: removing container" podUID="ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5" containerName="revoke-keys" Nov 03 00:35:29.948308 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:29.948214 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5" containerName="revoke-keys" Nov 03 00:35:29.948374 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:29.948346 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5" containerName="revoke-keys" Nov 03 00:35:29.951700 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:29.951677 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9zmbh" Nov 03 00:35:30.015549 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:30.015492 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9zmbh"] Nov 03 00:35:30.032602 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:30.032567 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50097625-1a31-4ab8-9d17-8901ebb130f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-9zmbh\" (UID: \"50097625-1a31-4ab8-9d17-8901ebb130f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9zmbh" Nov 03 00:35:30.032602 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:30.032608 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7kds\" (UniqueName: \"kubernetes.io/projected/50097625-1a31-4ab8-9d17-8901ebb130f8-kube-api-access-m7kds\") pod \"maas-api-revoke-sub-a4aef7d7-9zmbh\" (UID: \"50097625-1a31-4ab8-9d17-8901ebb130f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9zmbh" Nov 03 00:35:30.032875 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:30.032707 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50097625-1a31-4ab8-9d17-8901ebb130f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-9zmbh\" (UID: \"50097625-1a31-4ab8-9d17-8901ebb130f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9zmbh" Nov 03 00:35:30.051192 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:30.051161 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:35:30.051192 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:30.051185 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:35:30.133741 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:30.133702 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50097625-1a31-4ab8-9d17-8901ebb130f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-9zmbh\" (UID: \"50097625-1a31-4ab8-9d17-8901ebb130f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9zmbh" Nov 03 00:35:30.133958 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:30.133827 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50097625-1a31-4ab8-9d17-8901ebb130f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-9zmbh\" (UID: \"50097625-1a31-4ab8-9d17-8901ebb130f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9zmbh" Nov 03 00:35:30.133958 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:30.133866 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7kds\" (UniqueName: \"kubernetes.io/projected/50097625-1a31-4ab8-9d17-8901ebb130f8-kube-api-access-m7kds\") pod \"maas-api-revoke-sub-a4aef7d7-9zmbh\" (UID: \"50097625-1a31-4ab8-9d17-8901ebb130f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9zmbh" Nov 03 00:35:30.134498 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:30.134475 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50097625-1a31-4ab8-9d17-8901ebb130f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-9zmbh\" (UID: \"50097625-1a31-4ab8-9d17-8901ebb130f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9zmbh" Nov 03 00:35:30.136251 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:30.136228 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50097625-1a31-4ab8-9d17-8901ebb130f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-9zmbh\" (UID: \"50097625-1a31-4ab8-9d17-8901ebb130f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9zmbh" Nov 03 00:35:30.189383 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:30.189351 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7kds\" (UniqueName: \"kubernetes.io/projected/50097625-1a31-4ab8-9d17-8901ebb130f8-kube-api-access-m7kds\") pod \"maas-api-revoke-sub-a4aef7d7-9zmbh\" (UID: \"50097625-1a31-4ab8-9d17-8901ebb130f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9zmbh" Nov 03 00:35:30.265290 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:30.265204 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9zmbh" Nov 03 00:35:30.635412 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:30.635388 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9zmbh"] Nov 03 00:35:30.636998 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:35:30.636964 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50097625_1a31_4ab8_9d17_8901ebb130f8.slice/crio-65e03e18cbcf56625f88a29a9b0e291bf67fb9c5a43242af76502b0f049d409b WatchSource:0}: Error finding container 65e03e18cbcf56625f88a29a9b0e291bf67fb9c5a43242af76502b0f049d409b: Status 404 returned error can't find the container with id 65e03e18cbcf56625f88a29a9b0e291bf67fb9c5a43242af76502b0f049d409b Nov 03 00:35:31.617958 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:31.617927 5076 generic.go:356] "Generic (PLEG): container finished" podID="50097625-1a31-4ab8-9d17-8901ebb130f8" containerID="844c31e1f9c495fe1257f7e1326c1585727f396bb7340d55c23fc602b3311a6f" exitCode=0 Nov 03 00:35:31.618368 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:31.617975 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9zmbh" event={"ID":"50097625-1a31-4ab8-9d17-8901ebb130f8","Type":"ContainerDied","Data":"844c31e1f9c495fe1257f7e1326c1585727f396bb7340d55c23fc602b3311a6f"} Nov 03 00:35:31.618368 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:31.618010 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9zmbh" event={"ID":"50097625-1a31-4ab8-9d17-8901ebb130f8","Type":"ContainerStarted","Data":"65e03e18cbcf56625f88a29a9b0e291bf67fb9c5a43242af76502b0f049d409b"} Nov 03 00:35:32.771948 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:32.771920 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9zmbh" Nov 03 00:35:32.861330 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:32.861292 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50097625-1a31-4ab8-9d17-8901ebb130f8-kube-api-access-m7kds\" (UniqueName: \"kubernetes.io/projected/50097625-1a31-4ab8-9d17-8901ebb130f8-kube-api-access-m7kds\") pod \"50097625-1a31-4ab8-9d17-8901ebb130f8\" (UID: \"50097625-1a31-4ab8-9d17-8901ebb130f8\") " Nov 03 00:35:32.861518 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:32.861338 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50097625-1a31-4ab8-9d17-8901ebb130f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50097625-1a31-4ab8-9d17-8901ebb130f8-maas-api-openshift-ca\") pod \"50097625-1a31-4ab8-9d17-8901ebb130f8\" (UID: \"50097625-1a31-4ab8-9d17-8901ebb130f8\") " Nov 03 00:35:32.861518 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:32.861425 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50097625-1a31-4ab8-9d17-8901ebb130f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50097625-1a31-4ab8-9d17-8901ebb130f8-maas-api-xks-ca\") pod \"50097625-1a31-4ab8-9d17-8901ebb130f8\" (UID: \"50097625-1a31-4ab8-9d17-8901ebb130f8\") " Nov 03 00:35:32.861709 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:32.861677 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50097625-1a31-4ab8-9d17-8901ebb130f8-maas-api-openshift-ca" pod "50097625-1a31-4ab8-9d17-8901ebb130f8" (UID: "50097625-1a31-4ab8-9d17-8901ebb130f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:35:32.861946 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:32.861929 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50097625-1a31-4ab8-9d17-8901ebb130f8-maas-api-openshift-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:35:32.863420 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:32.863392 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50097625-1a31-4ab8-9d17-8901ebb130f8-maas-api-xks-ca" pod "50097625-1a31-4ab8-9d17-8901ebb130f8" (UID: "50097625-1a31-4ab8-9d17-8901ebb130f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:35:32.863515 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:32.863474 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50097625-1a31-4ab8-9d17-8901ebb130f8-kube-api-access-m7kds" pod "50097625-1a31-4ab8-9d17-8901ebb130f8" (UID: "50097625-1a31-4ab8-9d17-8901ebb130f8"). InnerVolumeSpecName "kube-api-access-m7kds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:35:32.962520 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:32.962486 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7kds\" (UniqueName: \"kubernetes.io/projected/50097625-1a31-4ab8-9d17-8901ebb130f8-kube-api-access-m7kds\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:35:32.962520 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:32.962517 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50097625-1a31-4ab8-9d17-8901ebb130f8-maas-api-xks-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:35:33.347140 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:33.347056 5076 scope.go:122] "RemoveContainer" containerID="8cb10fbe2d99767271ce0b4daf7f4f418d2ca33fa4d0d89cdf278ec1230d5d7e" Nov 03 00:35:33.355939 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:33.355914 5076 scope.go:122] "RemoveContainer" containerID="799763e3f06cbfd266fa9cd4b915a61413b1d6502ffe675b103ae80404910cbc" Nov 03 00:35:33.630570 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:33.630495 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9zmbh" Nov 03 00:35:33.630728 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:33.630494 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9zmbh" event={"ID":"50097625-1a31-4ab8-9d17-8901ebb130f8","Type":"ContainerDied","Data":"65e03e18cbcf56625f88a29a9b0e291bf67fb9c5a43242af76502b0f049d409b"} Nov 03 00:35:33.630728 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:33.630625 5076 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65e03e18cbcf56625f88a29a9b0e291bf67fb9c5a43242af76502b0f049d409b" Nov 03 00:35:39.883414 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:35:39.883372 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:35:51.067186 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:51.067143 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p8t9x"] Nov 03 00:35:51.068477 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:51.068450 5076 cpu_manager.go:414] "containerMap: removing container" podUID="50097625-1a31-4ab8-9d17-8901ebb130f8" containerName="revoke-keys" Nov 03 00:35:51.068667 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:51.068650 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50097625-1a31-4ab8-9d17-8901ebb130f8" containerName="revoke-keys" Nov 03 00:35:51.068953 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:51.068935 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="50097625-1a31-4ab8-9d17-8901ebb130f8" containerName="revoke-keys" Nov 03 00:35:51.093361 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:51.093315 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p8t9x" Nov 03 00:35:51.165500 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:51.165470 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:35:51.165948 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:51.165929 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:35:51.197398 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:51.197354 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/168923e9-9595-4a47-a0a5-1aba6d6c919c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-p8t9x\" (UID: \"168923e9-9595-4a47-a0a5-1aba6d6c919c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p8t9x" Nov 03 00:35:51.197823 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:51.197798 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-664cb\" (UniqueName: \"kubernetes.io/projected/168923e9-9595-4a47-a0a5-1aba6d6c919c-kube-api-access-664cb\") pod \"maas-api-revoke-sub-79805ca9-p8t9x\" (UID: \"168923e9-9595-4a47-a0a5-1aba6d6c919c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p8t9x" Nov 03 00:35:51.198037 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:51.198017 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/168923e9-9595-4a47-a0a5-1aba6d6c919c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-p8t9x\" (UID: \"168923e9-9595-4a47-a0a5-1aba6d6c919c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p8t9x" Nov 03 00:35:51.276917 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:51.265259 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p8t9x"] Nov 03 00:35:51.305967 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:51.305930 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/168923e9-9595-4a47-a0a5-1aba6d6c919c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-p8t9x\" (UID: \"168923e9-9595-4a47-a0a5-1aba6d6c919c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p8t9x" Nov 03 00:35:51.306323 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:51.306295 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-664cb\" (UniqueName: \"kubernetes.io/projected/168923e9-9595-4a47-a0a5-1aba6d6c919c-kube-api-access-664cb\") pod \"maas-api-revoke-sub-79805ca9-p8t9x\" (UID: \"168923e9-9595-4a47-a0a5-1aba6d6c919c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p8t9x" Nov 03 00:35:51.306496 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:51.306477 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/168923e9-9595-4a47-a0a5-1aba6d6c919c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-p8t9x\" (UID: \"168923e9-9595-4a47-a0a5-1aba6d6c919c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p8t9x" Nov 03 00:35:51.307513 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:51.307486 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/168923e9-9595-4a47-a0a5-1aba6d6c919c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-p8t9x\" (UID: \"168923e9-9595-4a47-a0a5-1aba6d6c919c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p8t9x" Nov 03 00:35:51.504148 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:51.504107 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-664cb\" (UniqueName: \"kubernetes.io/projected/168923e9-9595-4a47-a0a5-1aba6d6c919c-kube-api-access-664cb\") pod \"maas-api-revoke-sub-79805ca9-p8t9x\" (UID: \"168923e9-9595-4a47-a0a5-1aba6d6c919c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p8t9x" Nov 03 00:35:51.504642 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:51.504613 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/168923e9-9595-4a47-a0a5-1aba6d6c919c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-p8t9x\" (UID: \"168923e9-9595-4a47-a0a5-1aba6d6c919c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p8t9x" Nov 03 00:35:51.509121 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:51.509095 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p8t9x" Nov 03 00:35:52.423520 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:52.423493 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p8t9x"] Nov 03 00:35:52.425683 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:35:52.425655 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod168923e9_9595_4a47_a0a5_1aba6d6c919c.slice/crio-43e589e522eb00741b5f70986ebba4925132155bb91615dc181b2cdd8ff7c862 WatchSource:0}: Error finding container 43e589e522eb00741b5f70986ebba4925132155bb91615dc181b2cdd8ff7c862: Status 404 returned error can't find the container with id 43e589e522eb00741b5f70986ebba4925132155bb91615dc181b2cdd8ff7c862 Nov 03 00:35:52.781348 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:52.781305 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p8t9x" event={"ID":"168923e9-9595-4a47-a0a5-1aba6d6c919c","Type":"ContainerStarted","Data":"0293cbe20df172493e03e2610915e402d4e41adf31cf854176f69fba05bec9d3"} Nov 03 00:35:52.781525 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:52.781382 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p8t9x" event={"ID":"168923e9-9595-4a47-a0a5-1aba6d6c919c","Type":"ContainerStarted","Data":"43e589e522eb00741b5f70986ebba4925132155bb91615dc181b2cdd8ff7c862"} Nov 03 00:35:52.891681 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:52.891622 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p8t9x" podStartSLOduration=1.891607783 podStartE2EDuration="1.891607783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:52.889785218 +0000 UTC m=+6201.858317006" watchObservedRunningTime="2026-11-03 00:35:52.891607783 +0000 UTC m=+6201.860139567" Nov 03 00:35:53.787588 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:53.787549 5076 generic.go:356] "Generic (PLEG): container finished" podID="168923e9-9595-4a47-a0a5-1aba6d6c919c" containerID="0293cbe20df172493e03e2610915e402d4e41adf31cf854176f69fba05bec9d3" exitCode=0 Nov 03 00:35:53.788046 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:53.787631 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p8t9x" event={"ID":"168923e9-9595-4a47-a0a5-1aba6d6c919c","Type":"ContainerDied","Data":"0293cbe20df172493e03e2610915e402d4e41adf31cf854176f69fba05bec9d3"} Nov 03 00:35:54.938811 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:54.938785 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p8t9x" Nov 03 00:35:55.074543 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:55.074457 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/168923e9-9595-4a47-a0a5-1aba6d6c919c-kube-api-access-664cb\" (UniqueName: \"kubernetes.io/projected/168923e9-9595-4a47-a0a5-1aba6d6c919c-kube-api-access-664cb\") pod \"168923e9-9595-4a47-a0a5-1aba6d6c919c\" (UID: \"168923e9-9595-4a47-a0a5-1aba6d6c919c\") " Nov 03 00:35:55.074711 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:55.074660 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/168923e9-9595-4a47-a0a5-1aba6d6c919c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/168923e9-9595-4a47-a0a5-1aba6d6c919c-maas-api-xks-ca\") pod \"168923e9-9595-4a47-a0a5-1aba6d6c919c\" (UID: \"168923e9-9595-4a47-a0a5-1aba6d6c919c\") " Nov 03 00:35:55.074763 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:55.074721 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/168923e9-9595-4a47-a0a5-1aba6d6c919c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/168923e9-9595-4a47-a0a5-1aba6d6c919c-maas-api-openshift-ca\") pod \"168923e9-9595-4a47-a0a5-1aba6d6c919c\" (UID: \"168923e9-9595-4a47-a0a5-1aba6d6c919c\") " Nov 03 00:35:55.075178 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:55.075145 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/168923e9-9595-4a47-a0a5-1aba6d6c919c-maas-api-openshift-ca" pod "168923e9-9595-4a47-a0a5-1aba6d6c919c" (UID: "168923e9-9595-4a47-a0a5-1aba6d6c919c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:35:55.076686 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:55.076661 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/168923e9-9595-4a47-a0a5-1aba6d6c919c-maas-api-xks-ca" pod "168923e9-9595-4a47-a0a5-1aba6d6c919c" (UID: "168923e9-9595-4a47-a0a5-1aba6d6c919c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:35:55.076801 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:55.076775 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/168923e9-9595-4a47-a0a5-1aba6d6c919c-kube-api-access-664cb" pod "168923e9-9595-4a47-a0a5-1aba6d6c919c" (UID: "168923e9-9595-4a47-a0a5-1aba6d6c919c"). InnerVolumeSpecName "kube-api-access-664cb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:35:55.176360 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:55.176328 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/168923e9-9595-4a47-a0a5-1aba6d6c919c-maas-api-xks-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:35:55.176360 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:55.176357 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/168923e9-9595-4a47-a0a5-1aba6d6c919c-maas-api-openshift-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:35:55.176547 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:55.176368 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-664cb\" (UniqueName: \"kubernetes.io/projected/168923e9-9595-4a47-a0a5-1aba6d6c919c-kube-api-access-664cb\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:35:55.800001 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:55.799962 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p8t9x" event={"ID":"168923e9-9595-4a47-a0a5-1aba6d6c919c","Type":"ContainerDied","Data":"43e589e522eb00741b5f70986ebba4925132155bb91615dc181b2cdd8ff7c862"} Nov 03 00:35:55.800001 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:55.799982 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p8t9x" Nov 03 00:35:55.800001 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:35:55.800010 5076 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43e589e522eb00741b5f70986ebba4925132155bb91615dc181b2cdd8ff7c862" Nov 03 00:36:08.494914 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:08.487949 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-kjvs7"] Nov 03 00:36:08.494914 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:08.488651 5076 cpu_manager.go:414] "containerMap: removing container" podUID="168923e9-9595-4a47-a0a5-1aba6d6c919c" containerName="revoke-keys" Nov 03 00:36:08.494914 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:08.488671 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="168923e9-9595-4a47-a0a5-1aba6d6c919c" containerName="revoke-keys" Nov 03 00:36:08.494914 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:08.488828 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="168923e9-9595-4a47-a0a5-1aba6d6c919c" containerName="revoke-keys" Nov 03 00:36:08.494914 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:08.492821 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-kjvs7" Nov 03 00:36:08.539251 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:08.539217 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-kjvs7"] Nov 03 00:36:08.634914 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:08.611935 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:36:08.665912 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:08.618156 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:36:08.715556 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:08.645361 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5651ce1a-b668-48ff-95d2-f80078eb7863-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-kjvs7\" (UID: \"5651ce1a-b668-48ff-95d2-f80078eb7863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-kjvs7" Nov 03 00:36:08.715865 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:08.715839 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5651ce1a-b668-48ff-95d2-f80078eb7863-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-kjvs7\" (UID: \"5651ce1a-b668-48ff-95d2-f80078eb7863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-kjvs7" Nov 03 00:36:08.717148 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:08.717119 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slcvs\" (UniqueName: \"kubernetes.io/projected/5651ce1a-b668-48ff-95d2-f80078eb7863-kube-api-access-slcvs\") pod \"maas-api-revoke-sub-0b7e8edb-kjvs7\" (UID: \"5651ce1a-b668-48ff-95d2-f80078eb7863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-kjvs7" Nov 03 00:36:08.878009 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:08.876708 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5651ce1a-b668-48ff-95d2-f80078eb7863-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-kjvs7\" (UID: \"5651ce1a-b668-48ff-95d2-f80078eb7863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-kjvs7" Nov 03 00:36:08.878009 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:08.876776 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5651ce1a-b668-48ff-95d2-f80078eb7863-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-kjvs7\" (UID: \"5651ce1a-b668-48ff-95d2-f80078eb7863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-kjvs7" Nov 03 00:36:08.878009 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:08.876806 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slcvs\" (UniqueName: \"kubernetes.io/projected/5651ce1a-b668-48ff-95d2-f80078eb7863-kube-api-access-slcvs\") pod \"maas-api-revoke-sub-0b7e8edb-kjvs7\" (UID: \"5651ce1a-b668-48ff-95d2-f80078eb7863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-kjvs7" Nov 03 00:36:08.896913 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:08.889000 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5651ce1a-b668-48ff-95d2-f80078eb7863-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-kjvs7\" (UID: \"5651ce1a-b668-48ff-95d2-f80078eb7863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-kjvs7" Nov 03 00:36:08.928844 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:08.928812 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5651ce1a-b668-48ff-95d2-f80078eb7863-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-kjvs7\" (UID: \"5651ce1a-b668-48ff-95d2-f80078eb7863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-kjvs7" Nov 03 00:36:08.943020 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:08.942989 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slcvs\" (UniqueName: \"kubernetes.io/projected/5651ce1a-b668-48ff-95d2-f80078eb7863-kube-api-access-slcvs\") pod \"maas-api-revoke-sub-0b7e8edb-kjvs7\" (UID: \"5651ce1a-b668-48ff-95d2-f80078eb7863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-kjvs7" Nov 03 00:36:09.160171 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:09.159810 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-kjvs7" Nov 03 00:36:09.874131 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:09.874103 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-kjvs7"] Nov 03 00:36:09.877190 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:36:09.877163 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5651ce1a_b668_48ff_95d2_f80078eb7863.slice/crio-532f83d07679e843c4ebfa356cdd74cd88f09cbc0a21986b78d3bd6179ef8f62 WatchSource:0}: Error finding container 532f83d07679e843c4ebfa356cdd74cd88f09cbc0a21986b78d3bd6179ef8f62: Status 404 returned error can't find the container with id 532f83d07679e843c4ebfa356cdd74cd88f09cbc0a21986b78d3bd6179ef8f62 Nov 03 00:36:09.887955 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:36:09.887839 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:36:09.996110 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:09.996077 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-kjvs7" event={"ID":"5651ce1a-b668-48ff-95d2-f80078eb7863","Type":"ContainerStarted","Data":"532f83d07679e843c4ebfa356cdd74cd88f09cbc0a21986b78d3bd6179ef8f62"} Nov 03 00:36:11.003552 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:11.003510 5076 generic.go:356] "Generic (PLEG): container finished" podID="5651ce1a-b668-48ff-95d2-f80078eb7863" containerID="24dbc5940e7c01cbd41b32d684b1712bb3671dbda811e7f9794c5289cacb5cb5" exitCode=0 Nov 03 00:36:11.004042 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:11.003556 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-kjvs7" event={"ID":"5651ce1a-b668-48ff-95d2-f80078eb7863","Type":"ContainerDied","Data":"24dbc5940e7c01cbd41b32d684b1712bb3671dbda811e7f9794c5289cacb5cb5"} Nov 03 00:36:12.152515 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:12.152485 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-kjvs7" Nov 03 00:36:12.252734 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:12.252702 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5651ce1a-b668-48ff-95d2-f80078eb7863-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5651ce1a-b668-48ff-95d2-f80078eb7863-maas-api-openshift-ca\") pod \"5651ce1a-b668-48ff-95d2-f80078eb7863\" (UID: \"5651ce1a-b668-48ff-95d2-f80078eb7863\") " Nov 03 00:36:12.253169 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:12.253146 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5651ce1a-b668-48ff-95d2-f80078eb7863-maas-api-openshift-ca" pod "5651ce1a-b668-48ff-95d2-f80078eb7863" (UID: "5651ce1a-b668-48ff-95d2-f80078eb7863"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:36:12.353935 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:12.353837 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5651ce1a-b668-48ff-95d2-f80078eb7863-kube-api-access-slcvs\" (UniqueName: \"kubernetes.io/projected/5651ce1a-b668-48ff-95d2-f80078eb7863-kube-api-access-slcvs\") pod \"5651ce1a-b668-48ff-95d2-f80078eb7863\" (UID: \"5651ce1a-b668-48ff-95d2-f80078eb7863\") " Nov 03 00:36:12.353935 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:12.353898 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5651ce1a-b668-48ff-95d2-f80078eb7863-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5651ce1a-b668-48ff-95d2-f80078eb7863-maas-api-xks-ca\") pod \"5651ce1a-b668-48ff-95d2-f80078eb7863\" (UID: \"5651ce1a-b668-48ff-95d2-f80078eb7863\") " Nov 03 00:36:12.354295 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:12.354277 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5651ce1a-b668-48ff-95d2-f80078eb7863-maas-api-openshift-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:36:12.356048 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:12.356024 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5651ce1a-b668-48ff-95d2-f80078eb7863-maas-api-xks-ca" pod "5651ce1a-b668-48ff-95d2-f80078eb7863" (UID: "5651ce1a-b668-48ff-95d2-f80078eb7863"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:36:12.356153 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:12.356135 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5651ce1a-b668-48ff-95d2-f80078eb7863-kube-api-access-slcvs" pod "5651ce1a-b668-48ff-95d2-f80078eb7863" (UID: "5651ce1a-b668-48ff-95d2-f80078eb7863"). InnerVolumeSpecName "kube-api-access-slcvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:36:12.454821 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:12.454789 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slcvs\" (UniqueName: \"kubernetes.io/projected/5651ce1a-b668-48ff-95d2-f80078eb7863-kube-api-access-slcvs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:36:12.454821 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:12.454818 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5651ce1a-b668-48ff-95d2-f80078eb7863-maas-api-xks-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:36:13.017616 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:13.017581 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-kjvs7" event={"ID":"5651ce1a-b668-48ff-95d2-f80078eb7863","Type":"ContainerDied","Data":"532f83d07679e843c4ebfa356cdd74cd88f09cbc0a21986b78d3bd6179ef8f62"} Nov 03 00:36:13.017616 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:13.017602 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-kjvs7" Nov 03 00:36:13.017841 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:13.017630 5076 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="532f83d07679e843c4ebfa356cdd74cd88f09cbc0a21986b78d3bd6179ef8f62" Nov 03 00:36:19.838679 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:19.838639 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gz5p"] Nov 03 00:36:19.839322 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:19.839303 5076 cpu_manager.go:414] "containerMap: removing container" podUID="5651ce1a-b668-48ff-95d2-f80078eb7863" containerName="revoke-keys" Nov 03 00:36:19.839411 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:19.839325 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5651ce1a-b668-48ff-95d2-f80078eb7863" containerName="revoke-keys" Nov 03 00:36:19.839498 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:19.839486 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="5651ce1a-b668-48ff-95d2-f80078eb7863" containerName="revoke-keys" Nov 03 00:36:19.843367 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:19.843346 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gz5p" Nov 03 00:36:19.894621 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:19.894587 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gz5p"] Nov 03 00:36:19.926127 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:19.926095 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/263a8998-1c17-4e7e-8982-e7a5094d2951-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4gz5p\" (UID: \"263a8998-1c17-4e7e-8982-e7a5094d2951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gz5p" Nov 03 00:36:19.926298 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:19.926168 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pch7\" (UniqueName: \"kubernetes.io/projected/263a8998-1c17-4e7e-8982-e7a5094d2951-kube-api-access-7pch7\") pod \"maas-api-revoke-sub-3d4156f1-4gz5p\" (UID: \"263a8998-1c17-4e7e-8982-e7a5094d2951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gz5p" Nov 03 00:36:19.926388 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:19.926358 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/263a8998-1c17-4e7e-8982-e7a5094d2951-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4gz5p\" (UID: \"263a8998-1c17-4e7e-8982-e7a5094d2951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gz5p" Nov 03 00:36:19.938813 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:19.938785 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:36:19.938982 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:19.938853 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:36:20.027069 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:20.027032 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pch7\" (UniqueName: \"kubernetes.io/projected/263a8998-1c17-4e7e-8982-e7a5094d2951-kube-api-access-7pch7\") pod \"maas-api-revoke-sub-3d4156f1-4gz5p\" (UID: \"263a8998-1c17-4e7e-8982-e7a5094d2951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gz5p" Nov 03 00:36:20.027289 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:20.027134 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/263a8998-1c17-4e7e-8982-e7a5094d2951-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4gz5p\" (UID: \"263a8998-1c17-4e7e-8982-e7a5094d2951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gz5p" Nov 03 00:36:20.027289 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:20.027183 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/263a8998-1c17-4e7e-8982-e7a5094d2951-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4gz5p\" (UID: \"263a8998-1c17-4e7e-8982-e7a5094d2951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gz5p" Nov 03 00:36:20.027871 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:20.027844 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/263a8998-1c17-4e7e-8982-e7a5094d2951-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4gz5p\" (UID: \"263a8998-1c17-4e7e-8982-e7a5094d2951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gz5p" Nov 03 00:36:20.029453 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:20.029433 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/263a8998-1c17-4e7e-8982-e7a5094d2951-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4gz5p\" (UID: \"263a8998-1c17-4e7e-8982-e7a5094d2951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gz5p" Nov 03 00:36:20.082449 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:20.082410 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pch7\" (UniqueName: \"kubernetes.io/projected/263a8998-1c17-4e7e-8982-e7a5094d2951-kube-api-access-7pch7\") pod \"maas-api-revoke-sub-3d4156f1-4gz5p\" (UID: \"263a8998-1c17-4e7e-8982-e7a5094d2951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gz5p" Nov 03 00:36:20.156959 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:20.156874 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gz5p" Nov 03 00:36:20.521489 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:20.521464 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gz5p"] Nov 03 00:36:20.522428 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:36:20.522404 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod263a8998_1c17_4e7e_8982_e7a5094d2951.slice/crio-642d347bd9be213122fd94aeaf867e537b2a874619afee520b2ca8751ed12b47 WatchSource:0}: Error finding container 642d347bd9be213122fd94aeaf867e537b2a874619afee520b2ca8751ed12b47: Status 404 returned error can't find the container with id 642d347bd9be213122fd94aeaf867e537b2a874619afee520b2ca8751ed12b47 Nov 03 00:36:21.061851 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:21.061812 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gz5p" event={"ID":"263a8998-1c17-4e7e-8982-e7a5094d2951","Type":"ContainerStarted","Data":"d8afdfb3f2c610d307391ef6772ab086dfcef37327e1d3686bd2a269ec670313"} Nov 03 00:36:21.062279 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:21.061877 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gz5p" event={"ID":"263a8998-1c17-4e7e-8982-e7a5094d2951","Type":"ContainerStarted","Data":"642d347bd9be213122fd94aeaf867e537b2a874619afee520b2ca8751ed12b47"} Nov 03 00:36:21.172120 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:21.172061 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gz5p" podStartSLOduration=2.172044813 podStartE2EDuration="2.172044813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:36:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:36:21.170878061 +0000 UTC m=+6230.139409863" watchObservedRunningTime="2026-11-03 00:36:21.172044813 +0000 UTC m=+6230.140576598" Nov 03 00:36:22.068733 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:22.068701 5076 generic.go:356] "Generic (PLEG): container finished" podID="263a8998-1c17-4e7e-8982-e7a5094d2951" containerID="d8afdfb3f2c610d307391ef6772ab086dfcef37327e1d3686bd2a269ec670313" exitCode=0 Nov 03 00:36:22.069199 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:22.068791 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gz5p" event={"ID":"263a8998-1c17-4e7e-8982-e7a5094d2951","Type":"ContainerDied","Data":"d8afdfb3f2c610d307391ef6772ab086dfcef37327e1d3686bd2a269ec670313"} Nov 03 00:36:23.215822 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:23.215797 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gz5p" Nov 03 00:36:23.258703 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:23.258675 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/263a8998-1c17-4e7e-8982-e7a5094d2951-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/263a8998-1c17-4e7e-8982-e7a5094d2951-maas-api-xks-ca\") pod \"263a8998-1c17-4e7e-8982-e7a5094d2951\" (UID: \"263a8998-1c17-4e7e-8982-e7a5094d2951\") " Nov 03 00:36:23.258860 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:23.258742 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/263a8998-1c17-4e7e-8982-e7a5094d2951-kube-api-access-7pch7\" (UniqueName: \"kubernetes.io/projected/263a8998-1c17-4e7e-8982-e7a5094d2951-kube-api-access-7pch7\") pod \"263a8998-1c17-4e7e-8982-e7a5094d2951\" (UID: \"263a8998-1c17-4e7e-8982-e7a5094d2951\") " Nov 03 00:36:23.258860 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:23.258842 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/263a8998-1c17-4e7e-8982-e7a5094d2951-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/263a8998-1c17-4e7e-8982-e7a5094d2951-maas-api-openshift-ca\") pod \"263a8998-1c17-4e7e-8982-e7a5094d2951\" (UID: \"263a8998-1c17-4e7e-8982-e7a5094d2951\") " Nov 03 00:36:23.259242 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:23.259213 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/263a8998-1c17-4e7e-8982-e7a5094d2951-maas-api-openshift-ca" pod "263a8998-1c17-4e7e-8982-e7a5094d2951" (UID: "263a8998-1c17-4e7e-8982-e7a5094d2951"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:36:23.259658 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:23.259632 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/263a8998-1c17-4e7e-8982-e7a5094d2951-maas-api-openshift-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:36:23.260748 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:23.260729 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/263a8998-1c17-4e7e-8982-e7a5094d2951-maas-api-xks-ca" pod "263a8998-1c17-4e7e-8982-e7a5094d2951" (UID: "263a8998-1c17-4e7e-8982-e7a5094d2951"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:36:23.260870 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:23.260851 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/263a8998-1c17-4e7e-8982-e7a5094d2951-kube-api-access-7pch7" pod "263a8998-1c17-4e7e-8982-e7a5094d2951" (UID: "263a8998-1c17-4e7e-8982-e7a5094d2951"). InnerVolumeSpecName "kube-api-access-7pch7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:36:23.360857 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:23.360780 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/263a8998-1c17-4e7e-8982-e7a5094d2951-maas-api-xks-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:36:23.360857 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:23.360808 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pch7\" (UniqueName: \"kubernetes.io/projected/263a8998-1c17-4e7e-8982-e7a5094d2951-kube-api-access-7pch7\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:36:24.089049 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:24.089016 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gz5p" Nov 03 00:36:24.089049 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:24.089032 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gz5p" event={"ID":"263a8998-1c17-4e7e-8982-e7a5094d2951","Type":"ContainerDied","Data":"642d347bd9be213122fd94aeaf867e537b2a874619afee520b2ca8751ed12b47"} Nov 03 00:36:24.089296 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:24.089103 5076 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="642d347bd9be213122fd94aeaf867e537b2a874619afee520b2ca8751ed12b47" Nov 03 00:36:33.380228 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:33.380195 5076 scope.go:122] "RemoveContainer" containerID="8cf0c1f898acf597eb56e74847955de4f7905639b1aa8ed7a5d361db5647808c" Nov 03 00:36:38.690285 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:38.690246 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gz5p"] Nov 03 00:36:38.737879 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:38.737847 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4gz5p"] Nov 03 00:36:38.813022 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:38.812978 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnht2"] Nov 03 00:36:38.813664 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:38.813640 5076 cpu_manager.go:414] "containerMap: removing container" podUID="263a8998-1c17-4e7e-8982-e7a5094d2951" containerName="revoke-keys" Nov 03 00:36:38.813792 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:38.813665 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="263a8998-1c17-4e7e-8982-e7a5094d2951" containerName="revoke-keys" Nov 03 00:36:38.813792 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:38.813771 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="263a8998-1c17-4e7e-8982-e7a5094d2951" containerName="revoke-keys" Nov 03 00:36:38.818772 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:38.818745 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnht2" Nov 03 00:36:38.869226 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:38.869192 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:36:38.869474 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:38.869235 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:36:38.870095 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:38.870071 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnht2"] Nov 03 00:36:38.911462 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:38.911426 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rnht2\" (UID: \"282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnht2" Nov 03 00:36:38.911656 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:38.911525 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phhrg\" (UniqueName: \"kubernetes.io/projected/282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30-kube-api-access-phhrg\") pod \"maas-api-revoke-sub-3d4156f1-rnht2\" (UID: \"282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnht2" Nov 03 00:36:38.911713 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:38.911677 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rnht2\" (UID: \"282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnht2" Nov 03 00:36:39.013386 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:39.013297 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rnht2\" (UID: \"282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnht2" Nov 03 00:36:39.013386 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:39.013351 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phhrg\" (UniqueName: \"kubernetes.io/projected/282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30-kube-api-access-phhrg\") pod \"maas-api-revoke-sub-3d4156f1-rnht2\" (UID: \"282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnht2" Nov 03 00:36:39.013634 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:39.013436 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rnht2\" (UID: \"282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnht2" Nov 03 00:36:39.014070 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:39.014040 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rnht2\" (UID: \"282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnht2" Nov 03 00:36:39.015838 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:39.015816 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rnht2\" (UID: \"282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnht2" Nov 03 00:36:39.067697 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:39.067667 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phhrg\" (UniqueName: \"kubernetes.io/projected/282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30-kube-api-access-phhrg\") pod \"maas-api-revoke-sub-3d4156f1-rnht2\" (UID: \"282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnht2" Nov 03 00:36:39.132403 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:39.132363 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnht2" Nov 03 00:36:39.150635 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:39.150597 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="263a8998-1c17-4e7e-8982-e7a5094d2951" path="/var/lib/kubelet/pods/263a8998-1c17-4e7e-8982-e7a5094d2951/volumes" Nov 03 00:36:39.500701 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:39.500669 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnht2"] Nov 03 00:36:39.501984 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:36:39.501957 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod282fbaca_2dd8_4d0d_ab5d_c3bcb50f8d30.slice/crio-051a16a38f6307a5e3403405b30f14498968e3f103317089a5eb869b52d6cb73 WatchSource:0}: Error finding container 051a16a38f6307a5e3403405b30f14498968e3f103317089a5eb869b52d6cb73: Status 404 returned error can't find the container with id 051a16a38f6307a5e3403405b30f14498968e3f103317089a5eb869b52d6cb73 Nov 03 00:36:39.883203 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:36:39.883164 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:36:40.204779 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:40.204744 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnht2" event={"ID":"282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30","Type":"ContainerStarted","Data":"cbb3a609868c7d0831abcf3506f7fcc4eff82f2c0074287de812cb3a8e957795"} Nov 03 00:36:40.204979 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:40.204796 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnht2" event={"ID":"282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30","Type":"ContainerStarted","Data":"051a16a38f6307a5e3403405b30f14498968e3f103317089a5eb869b52d6cb73"} Nov 03 00:36:40.361356 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:40.361295 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnht2" podStartSLOduration=2.361280432 podStartE2EDuration="2.361280432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:36:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:36:40.3126911 +0000 UTC m=+6249.281222884" watchObservedRunningTime="2026-11-03 00:36:40.361280432 +0000 UTC m=+6249.329812216" Nov 03 00:36:41.212682 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:41.212598 5076 generic.go:356] "Generic (PLEG): container finished" podID="282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30" containerID="cbb3a609868c7d0831abcf3506f7fcc4eff82f2c0074287de812cb3a8e957795" exitCode=0 Nov 03 00:36:41.213134 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:41.212680 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnht2" event={"ID":"282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30","Type":"ContainerDied","Data":"cbb3a609868c7d0831abcf3506f7fcc4eff82f2c0074287de812cb3a8e957795"} Nov 03 00:36:42.387878 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:42.387854 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnht2" Nov 03 00:36:42.448662 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:42.448632 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30-maas-api-xks-ca\") pod \"282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30\" (UID: \"282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30\") " Nov 03 00:36:42.448832 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:42.448678 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30-maas-api-openshift-ca\") pod \"282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30\" (UID: \"282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30\") " Nov 03 00:36:42.449075 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:42.449051 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30-maas-api-openshift-ca" pod "282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30" (UID: "282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:36:42.450695 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:42.450674 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30-maas-api-xks-ca" pod "282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30" (UID: "282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:36:42.549317 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:42.549227 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30-kube-api-access-phhrg\" (UniqueName: \"kubernetes.io/projected/282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30-kube-api-access-phhrg\") pod \"282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30\" (UID: \"282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30\") " Nov 03 00:36:42.549500 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:42.549476 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30-maas-api-xks-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:36:42.549500 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:42.549489 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30-maas-api-openshift-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:36:42.551434 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:42.551407 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30-kube-api-access-phhrg" pod "282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30" (UID: "282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30"). InnerVolumeSpecName "kube-api-access-phhrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:36:42.650159 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:42.650119 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phhrg\" (UniqueName: \"kubernetes.io/projected/282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30-kube-api-access-phhrg\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:36:43.239711 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:43.239592 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnht2" event={"ID":"282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30","Type":"ContainerDied","Data":"051a16a38f6307a5e3403405b30f14498968e3f103317089a5eb869b52d6cb73"} Nov 03 00:36:43.239711 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:43.239630 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnht2" Nov 03 00:36:43.239711 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:36:43.239674 5076 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="051a16a38f6307a5e3403405b30f14498968e3f103317089a5eb869b52d6cb73" Nov 03 00:37:09.885219 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:37:09.885177 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:37:16.033931 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:16.033872 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7cfh8"] Nov 03 00:37:16.034595 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:16.034573 5076 cpu_manager.go:414] "containerMap: removing container" podUID="282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30" containerName="revoke-keys" Nov 03 00:37:16.034595 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:16.034595 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30" containerName="revoke-keys" Nov 03 00:37:16.034785 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:16.034756 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30" containerName="revoke-keys" Nov 03 00:37:16.038208 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:16.038187 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7cfh8" Nov 03 00:37:16.091234 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:16.091202 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7cfh8"] Nov 03 00:37:16.135026 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:16.134991 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:37:16.135193 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:16.135045 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:37:16.201916 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:16.201868 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/455a916c-ca5a-4495-b27a-7d51d7a64118-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-7cfh8\" (UID: \"455a916c-ca5a-4495-b27a-7d51d7a64118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7cfh8" Nov 03 00:37:16.202101 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:16.201956 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h4ql\" (UniqueName: \"kubernetes.io/projected/455a916c-ca5a-4495-b27a-7d51d7a64118-kube-api-access-4h4ql\") pod \"maas-api-revoke-sub-e2fc9c5b-7cfh8\" (UID: \"455a916c-ca5a-4495-b27a-7d51d7a64118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7cfh8" Nov 03 00:37:16.202101 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:16.202008 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/455a916c-ca5a-4495-b27a-7d51d7a64118-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-7cfh8\" (UID: \"455a916c-ca5a-4495-b27a-7d51d7a64118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7cfh8" Nov 03 00:37:16.303132 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:16.303046 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/455a916c-ca5a-4495-b27a-7d51d7a64118-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-7cfh8\" (UID: \"455a916c-ca5a-4495-b27a-7d51d7a64118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7cfh8" Nov 03 00:37:16.303132 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:16.303084 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h4ql\" (UniqueName: \"kubernetes.io/projected/455a916c-ca5a-4495-b27a-7d51d7a64118-kube-api-access-4h4ql\") pod \"maas-api-revoke-sub-e2fc9c5b-7cfh8\" (UID: \"455a916c-ca5a-4495-b27a-7d51d7a64118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7cfh8" Nov 03 00:37:16.303382 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:16.303135 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/455a916c-ca5a-4495-b27a-7d51d7a64118-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-7cfh8\" (UID: \"455a916c-ca5a-4495-b27a-7d51d7a64118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7cfh8" Nov 03 00:37:16.303797 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:16.303777 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/455a916c-ca5a-4495-b27a-7d51d7a64118-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-7cfh8\" (UID: \"455a916c-ca5a-4495-b27a-7d51d7a64118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7cfh8" Nov 03 00:37:16.305471 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:16.305455 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/455a916c-ca5a-4495-b27a-7d51d7a64118-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-7cfh8\" (UID: \"455a916c-ca5a-4495-b27a-7d51d7a64118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7cfh8" Nov 03 00:37:16.357514 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:16.357475 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h4ql\" (UniqueName: \"kubernetes.io/projected/455a916c-ca5a-4495-b27a-7d51d7a64118-kube-api-access-4h4ql\") pod \"maas-api-revoke-sub-e2fc9c5b-7cfh8\" (UID: \"455a916c-ca5a-4495-b27a-7d51d7a64118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7cfh8" Nov 03 00:37:16.652521 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:16.652431 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7cfh8" Nov 03 00:37:17.016770 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:17.016737 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7cfh8"] Nov 03 00:37:17.017935 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:37:17.017882 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod455a916c_ca5a_4495_b27a_7d51d7a64118.slice/crio-40b57e8fc4ab6cad8152a3090ec1cc3110d127e6f4bd02b376f774b13d044422 WatchSource:0}: Error finding container 40b57e8fc4ab6cad8152a3090ec1cc3110d127e6f4bd02b376f774b13d044422: Status 404 returned error can't find the container with id 40b57e8fc4ab6cad8152a3090ec1cc3110d127e6f4bd02b376f774b13d044422 Nov 03 00:37:17.630475 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:17.630390 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7cfh8" event={"ID":"455a916c-ca5a-4495-b27a-7d51d7a64118","Type":"ContainerStarted","Data":"dddb10365ae1884a87611bd20df6c8cb1b06bd8b47d9aec28aa29b6ce4fb7eea"} Nov 03 00:37:17.630475 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:17.630451 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7cfh8" event={"ID":"455a916c-ca5a-4495-b27a-7d51d7a64118","Type":"ContainerStarted","Data":"40b57e8fc4ab6cad8152a3090ec1cc3110d127e6f4bd02b376f774b13d044422"} Nov 03 00:37:17.740124 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:17.740073 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7cfh8" podStartSLOduration=2.7400578920000003 podStartE2EDuration="2.740057892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:37:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:37:17.7393588 +0000 UTC m=+6286.707890583" watchObservedRunningTime="2026-11-03 00:37:17.740057892 +0000 UTC m=+6286.708589675" Nov 03 00:37:18.636954 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:18.636922 5076 generic.go:356] "Generic (PLEG): container finished" podID="455a916c-ca5a-4495-b27a-7d51d7a64118" containerID="dddb10365ae1884a87611bd20df6c8cb1b06bd8b47d9aec28aa29b6ce4fb7eea" exitCode=0 Nov 03 00:37:18.637444 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:18.637058 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7cfh8" event={"ID":"455a916c-ca5a-4495-b27a-7d51d7a64118","Type":"ContainerDied","Data":"dddb10365ae1884a87611bd20df6c8cb1b06bd8b47d9aec28aa29b6ce4fb7eea"} Nov 03 00:37:19.786718 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:19.786691 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7cfh8" Nov 03 00:37:19.942996 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:19.942963 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/455a916c-ca5a-4495-b27a-7d51d7a64118-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/455a916c-ca5a-4495-b27a-7d51d7a64118-maas-api-openshift-ca\") pod \"455a916c-ca5a-4495-b27a-7d51d7a64118\" (UID: \"455a916c-ca5a-4495-b27a-7d51d7a64118\") " Nov 03 00:37:19.943202 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:19.943013 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/455a916c-ca5a-4495-b27a-7d51d7a64118-kube-api-access-4h4ql\" (UniqueName: \"kubernetes.io/projected/455a916c-ca5a-4495-b27a-7d51d7a64118-kube-api-access-4h4ql\") pod \"455a916c-ca5a-4495-b27a-7d51d7a64118\" (UID: \"455a916c-ca5a-4495-b27a-7d51d7a64118\") " Nov 03 00:37:19.943202 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:19.943045 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/455a916c-ca5a-4495-b27a-7d51d7a64118-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/455a916c-ca5a-4495-b27a-7d51d7a64118-maas-api-xks-ca\") pod \"455a916c-ca5a-4495-b27a-7d51d7a64118\" (UID: \"455a916c-ca5a-4495-b27a-7d51d7a64118\") " Nov 03 00:37:19.943336 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:19.943313 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/455a916c-ca5a-4495-b27a-7d51d7a64118-maas-api-openshift-ca" pod "455a916c-ca5a-4495-b27a-7d51d7a64118" (UID: "455a916c-ca5a-4495-b27a-7d51d7a64118"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:37:19.943426 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:19.943403 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/455a916c-ca5a-4495-b27a-7d51d7a64118-maas-api-openshift-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:37:19.945125 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:19.945101 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/455a916c-ca5a-4495-b27a-7d51d7a64118-kube-api-access-4h4ql" pod "455a916c-ca5a-4495-b27a-7d51d7a64118" (UID: "455a916c-ca5a-4495-b27a-7d51d7a64118"). InnerVolumeSpecName "kube-api-access-4h4ql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:37:19.945125 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:19.945113 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/455a916c-ca5a-4495-b27a-7d51d7a64118-maas-api-xks-ca" pod "455a916c-ca5a-4495-b27a-7d51d7a64118" (UID: "455a916c-ca5a-4495-b27a-7d51d7a64118"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:37:20.044381 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:20.044337 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h4ql\" (UniqueName: \"kubernetes.io/projected/455a916c-ca5a-4495-b27a-7d51d7a64118-kube-api-access-4h4ql\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:37:20.044381 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:20.044382 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/455a916c-ca5a-4495-b27a-7d51d7a64118-maas-api-xks-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:37:20.649008 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:20.648977 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7cfh8" Nov 03 00:37:20.649008 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:20.648994 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7cfh8" event={"ID":"455a916c-ca5a-4495-b27a-7d51d7a64118","Type":"ContainerDied","Data":"40b57e8fc4ab6cad8152a3090ec1cc3110d127e6f4bd02b376f774b13d044422"} Nov 03 00:37:20.649284 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:20.649039 5076 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40b57e8fc4ab6cad8152a3090ec1cc3110d127e6f4bd02b376f774b13d044422" Nov 03 00:37:26.805993 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:26.805955 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-f2ddv"] Nov 03 00:37:26.806672 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:26.806650 5076 cpu_manager.go:414] "containerMap: removing container" podUID="455a916c-ca5a-4495-b27a-7d51d7a64118" containerName="revoke-keys" Nov 03 00:37:26.806796 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:26.806674 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="455a916c-ca5a-4495-b27a-7d51d7a64118" containerName="revoke-keys" Nov 03 00:37:26.806864 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:26.806794 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="455a916c-ca5a-4495-b27a-7d51d7a64118" containerName="revoke-keys" Nov 03 00:37:26.810085 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:26.810062 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-f2ddv" Nov 03 00:37:26.907578 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:26.907527 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cc0827f-b2ba-4586-bcc4-d07ded683156-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-f2ddv\" (UID: \"6cc0827f-b2ba-4586-bcc4-d07ded683156\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-f2ddv" Nov 03 00:37:26.907762 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:26.907586 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4w47\" (UniqueName: \"kubernetes.io/projected/6cc0827f-b2ba-4586-bcc4-d07ded683156-kube-api-access-k4w47\") pod \"maas-api-revoke-sub-5a29efb3-f2ddv\" (UID: \"6cc0827f-b2ba-4586-bcc4-d07ded683156\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-f2ddv" Nov 03 00:37:26.907762 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:26.907712 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cc0827f-b2ba-4586-bcc4-d07ded683156-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-f2ddv\" (UID: \"6cc0827f-b2ba-4586-bcc4-d07ded683156\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-f2ddv" Nov 03 00:37:26.910621 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:26.910596 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:37:26.910727 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:26.910647 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:37:26.913290 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:26.913270 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-f2ddv"] Nov 03 00:37:27.008801 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:27.008767 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cc0827f-b2ba-4586-bcc4-d07ded683156-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-f2ddv\" (UID: \"6cc0827f-b2ba-4586-bcc4-d07ded683156\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-f2ddv" Nov 03 00:37:27.008801 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:27.008807 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4w47\" (UniqueName: \"kubernetes.io/projected/6cc0827f-b2ba-4586-bcc4-d07ded683156-kube-api-access-k4w47\") pod \"maas-api-revoke-sub-5a29efb3-f2ddv\" (UID: \"6cc0827f-b2ba-4586-bcc4-d07ded683156\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-f2ddv" Nov 03 00:37:27.009073 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:27.008969 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cc0827f-b2ba-4586-bcc4-d07ded683156-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-f2ddv\" (UID: \"6cc0827f-b2ba-4586-bcc4-d07ded683156\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-f2ddv" Nov 03 00:37:27.009396 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:27.009378 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cc0827f-b2ba-4586-bcc4-d07ded683156-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-f2ddv\" (UID: \"6cc0827f-b2ba-4586-bcc4-d07ded683156\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-f2ddv" Nov 03 00:37:27.011226 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:27.011195 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cc0827f-b2ba-4586-bcc4-d07ded683156-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-f2ddv\" (UID: \"6cc0827f-b2ba-4586-bcc4-d07ded683156\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-f2ddv" Nov 03 00:37:27.063874 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:27.063793 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4w47\" (UniqueName: \"kubernetes.io/projected/6cc0827f-b2ba-4586-bcc4-d07ded683156-kube-api-access-k4w47\") pod \"maas-api-revoke-sub-5a29efb3-f2ddv\" (UID: \"6cc0827f-b2ba-4586-bcc4-d07ded683156\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-f2ddv" Nov 03 00:37:27.122850 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:27.122810 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-f2ddv" Nov 03 00:37:27.489301 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:27.489272 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-f2ddv"] Nov 03 00:37:27.490591 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:37:27.490565 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cc0827f_b2ba_4586_bcc4_d07ded683156.slice/crio-4847dc9acd1e542508d2186304808cfe638952f9ff7413c1f3b0652a4bc57a1c WatchSource:0}: Error finding container 4847dc9acd1e542508d2186304808cfe638952f9ff7413c1f3b0652a4bc57a1c: Status 404 returned error can't find the container with id 4847dc9acd1e542508d2186304808cfe638952f9ff7413c1f3b0652a4bc57a1c Nov 03 00:37:27.689333 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:27.689296 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-f2ddv" event={"ID":"6cc0827f-b2ba-4586-bcc4-d07ded683156","Type":"ContainerStarted","Data":"4847dc9acd1e542508d2186304808cfe638952f9ff7413c1f3b0652a4bc57a1c"} Nov 03 00:37:28.695788 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:28.695749 5076 generic.go:356] "Generic (PLEG): container finished" podID="6cc0827f-b2ba-4586-bcc4-d07ded683156" containerID="fe2327dce914d6b1a9a676ed95df3b7da5e5e5d56ec22be3766d2c52235b7b3c" exitCode=0 Nov 03 00:37:28.696245 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:28.695843 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-f2ddv" event={"ID":"6cc0827f-b2ba-4586-bcc4-d07ded683156","Type":"ContainerDied","Data":"fe2327dce914d6b1a9a676ed95df3b7da5e5e5d56ec22be3766d2c52235b7b3c"} Nov 03 00:37:29.923085 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:29.923052 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-f2ddv" Nov 03 00:37:30.070765 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:30.070670 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cc0827f-b2ba-4586-bcc4-d07ded683156-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cc0827f-b2ba-4586-bcc4-d07ded683156-maas-api-openshift-ca\") pod \"6cc0827f-b2ba-4586-bcc4-d07ded683156\" (UID: \"6cc0827f-b2ba-4586-bcc4-d07ded683156\") " Nov 03 00:37:30.071589 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:30.071403 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cc0827f-b2ba-4586-bcc4-d07ded683156-maas-api-openshift-ca" pod "6cc0827f-b2ba-4586-bcc4-d07ded683156" (UID: "6cc0827f-b2ba-4586-bcc4-d07ded683156"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:37:30.071798 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:30.071777 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cc0827f-b2ba-4586-bcc4-d07ded683156-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cc0827f-b2ba-4586-bcc4-d07ded683156-maas-api-xks-ca\") pod \"6cc0827f-b2ba-4586-bcc4-d07ded683156\" (UID: \"6cc0827f-b2ba-4586-bcc4-d07ded683156\") " Nov 03 00:37:30.072477 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:30.072447 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cc0827f-b2ba-4586-bcc4-d07ded683156-kube-api-access-k4w47\" (UniqueName: \"kubernetes.io/projected/6cc0827f-b2ba-4586-bcc4-d07ded683156-kube-api-access-k4w47\") pod \"6cc0827f-b2ba-4586-bcc4-d07ded683156\" (UID: \"6cc0827f-b2ba-4586-bcc4-d07ded683156\") " Nov 03 00:37:30.073496 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:30.073474 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cc0827f-b2ba-4586-bcc4-d07ded683156-maas-api-openshift-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:37:30.074800 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:30.074772 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cc0827f-b2ba-4586-bcc4-d07ded683156-maas-api-xks-ca" pod "6cc0827f-b2ba-4586-bcc4-d07ded683156" (UID: "6cc0827f-b2ba-4586-bcc4-d07ded683156"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:37:30.076605 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:30.076579 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cc0827f-b2ba-4586-bcc4-d07ded683156-kube-api-access-k4w47" pod "6cc0827f-b2ba-4586-bcc4-d07ded683156" (UID: "6cc0827f-b2ba-4586-bcc4-d07ded683156"). InnerVolumeSpecName "kube-api-access-k4w47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:37:30.175137 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:30.175095 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cc0827f-b2ba-4586-bcc4-d07ded683156-maas-api-xks-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:37:30.175137 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:30.175139 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4w47\" (UniqueName: \"kubernetes.io/projected/6cc0827f-b2ba-4586-bcc4-d07ded683156-kube-api-access-k4w47\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:37:30.742914 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:30.742121 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-f2ddv" event={"ID":"6cc0827f-b2ba-4586-bcc4-d07ded683156","Type":"ContainerDied","Data":"4847dc9acd1e542508d2186304808cfe638952f9ff7413c1f3b0652a4bc57a1c"} Nov 03 00:37:30.742914 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:30.742189 5076 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4847dc9acd1e542508d2186304808cfe638952f9ff7413c1f3b0652a4bc57a1c" Nov 03 00:37:30.742914 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:30.742261 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-f2ddv" Nov 03 00:37:31.463216 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:31.463163 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gctbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:37:31.465548 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:31.465493 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" podUID="d6db6fe8-b76c-4878-b0f3-bd347b2d0cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:37:31.820057 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:31.819934 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-784vq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:37:31.820352 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:31.820306 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" podUID="506655af-c50a-4207-906e-9f705fc8f9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:37:32.329680 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:32.329648 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 03 00:37:32.423914 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:32.423873 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 03 00:37:39.883453 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:37:39.883415 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:37:42.044036 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:42.043997 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wqw5q"] Nov 03 00:37:42.091561 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:42.091509 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wqw5q"] Nov 03 00:37:43.146800 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:43.146767 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e3a97b9-b122-48e9-97c4-15b1a0e43e26" path="/var/lib/kubelet/pods/5e3a97b9-b122-48e9-97c4-15b1a0e43e26/volumes" Nov 03 00:37:45.669686 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:45.669646 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnht2"] Nov 03 00:37:45.717049 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:45.717012 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rnht2"] Nov 03 00:37:45.793913 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:45.793871 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4pdpj"] Nov 03 00:37:45.794330 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:45.794316 5076 cpu_manager.go:414] "containerMap: removing container" podUID="6cc0827f-b2ba-4586-bcc4-d07ded683156" containerName="revoke-keys" Nov 03 00:37:45.794379 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:45.794331 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cc0827f-b2ba-4586-bcc4-d07ded683156" containerName="revoke-keys" Nov 03 00:37:45.794416 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:45.794411 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cc0827f-b2ba-4586-bcc4-d07ded683156" containerName="revoke-keys" Nov 03 00:37:45.799006 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:45.798985 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4pdpj" Nov 03 00:37:45.850933 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:45.850902 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4pdpj"] Nov 03 00:37:45.854207 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:45.854181 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:37:45.854358 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:45.854181 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:37:45.878351 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:45.878319 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85b07127-3904-4eb7-ae74-20456b229947-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4pdpj\" (UID: \"85b07127-3904-4eb7-ae74-20456b229947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4pdpj" Nov 03 00:37:45.878547 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:45.878363 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85b07127-3904-4eb7-ae74-20456b229947-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4pdpj\" (UID: \"85b07127-3904-4eb7-ae74-20456b229947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4pdpj" Nov 03 00:37:45.878547 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:45.878497 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6nlg\" (UniqueName: \"kubernetes.io/projected/85b07127-3904-4eb7-ae74-20456b229947-kube-api-access-l6nlg\") pod \"maas-api-revoke-sub-3d4156f1-4pdpj\" (UID: \"85b07127-3904-4eb7-ae74-20456b229947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4pdpj" Nov 03 00:37:45.979192 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:45.979157 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85b07127-3904-4eb7-ae74-20456b229947-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4pdpj\" (UID: \"85b07127-3904-4eb7-ae74-20456b229947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4pdpj" Nov 03 00:37:45.979399 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:45.979244 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6nlg\" (UniqueName: \"kubernetes.io/projected/85b07127-3904-4eb7-ae74-20456b229947-kube-api-access-l6nlg\") pod \"maas-api-revoke-sub-3d4156f1-4pdpj\" (UID: \"85b07127-3904-4eb7-ae74-20456b229947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4pdpj" Nov 03 00:37:45.979399 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:45.979301 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85b07127-3904-4eb7-ae74-20456b229947-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4pdpj\" (UID: \"85b07127-3904-4eb7-ae74-20456b229947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4pdpj" Nov 03 00:37:45.979840 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:45.979812 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85b07127-3904-4eb7-ae74-20456b229947-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4pdpj\" (UID: \"85b07127-3904-4eb7-ae74-20456b229947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4pdpj" Nov 03 00:37:45.981533 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:45.981513 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85b07127-3904-4eb7-ae74-20456b229947-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4pdpj\" (UID: \"85b07127-3904-4eb7-ae74-20456b229947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4pdpj" Nov 03 00:37:46.033732 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:46.033699 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6nlg\" (UniqueName: \"kubernetes.io/projected/85b07127-3904-4eb7-ae74-20456b229947-kube-api-access-l6nlg\") pod \"maas-api-revoke-sub-3d4156f1-4pdpj\" (UID: \"85b07127-3904-4eb7-ae74-20456b229947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4pdpj" Nov 03 00:37:46.111927 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:46.111880 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4pdpj" Nov 03 00:37:46.477343 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:46.477314 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4pdpj"] Nov 03 00:37:46.479134 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:37:46.479104 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85b07127_3904_4eb7_ae74_20456b229947.slice/crio-68ca2d2ee1608c4e6e30db580913094cbe5056e075a7f9b57ce2aa3f65c84b80 WatchSource:0}: Error finding container 68ca2d2ee1608c4e6e30db580913094cbe5056e075a7f9b57ce2aa3f65c84b80: Status 404 returned error can't find the container with id 68ca2d2ee1608c4e6e30db580913094cbe5056e075a7f9b57ce2aa3f65c84b80 Nov 03 00:37:46.847588 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:46.847543 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4pdpj" event={"ID":"85b07127-3904-4eb7-ae74-20456b229947","Type":"ContainerStarted","Data":"906b7c4409ccf54850d6cf474171b7f747c4a3669732da9ba000d5ae33d52bee"} Nov 03 00:37:46.848013 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:46.847632 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4pdpj" event={"ID":"85b07127-3904-4eb7-ae74-20456b229947","Type":"ContainerStarted","Data":"68ca2d2ee1608c4e6e30db580913094cbe5056e075a7f9b57ce2aa3f65c84b80"} Nov 03 00:37:46.958093 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:46.958034 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4pdpj" podStartSLOduration=1.9580183660000001 podStartE2EDuration="1.958018366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:37:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:37:46.956653736 +0000 UTC m=+6315.925185521" watchObservedRunningTime="2026-11-03 00:37:46.958018366 +0000 UTC m=+6315.926550163" Nov 03 00:37:47.157256 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:47.157175 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30" path="/var/lib/kubelet/pods/282fbaca-2dd8-4d0d-ab5d-c3bcb50f8d30/volumes" Nov 03 00:37:47.856015 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:47.855982 5076 generic.go:356] "Generic (PLEG): container finished" podID="85b07127-3904-4eb7-ae74-20456b229947" containerID="906b7c4409ccf54850d6cf474171b7f747c4a3669732da9ba000d5ae33d52bee" exitCode=0 Nov 03 00:37:47.856433 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:47.856029 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4pdpj" event={"ID":"85b07127-3904-4eb7-ae74-20456b229947","Type":"ContainerDied","Data":"906b7c4409ccf54850d6cf474171b7f747c4a3669732da9ba000d5ae33d52bee"} Nov 03 00:37:49.003258 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:49.003229 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4pdpj" Nov 03 00:37:49.110527 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:49.110493 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85b07127-3904-4eb7-ae74-20456b229947-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85b07127-3904-4eb7-ae74-20456b229947-maas-api-xks-ca\") pod \"85b07127-3904-4eb7-ae74-20456b229947\" (UID: \"85b07127-3904-4eb7-ae74-20456b229947\") " Nov 03 00:37:49.110706 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:49.110599 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85b07127-3904-4eb7-ae74-20456b229947-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85b07127-3904-4eb7-ae74-20456b229947-maas-api-openshift-ca\") pod \"85b07127-3904-4eb7-ae74-20456b229947\" (UID: \"85b07127-3904-4eb7-ae74-20456b229947\") " Nov 03 00:37:49.110706 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:49.110628 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85b07127-3904-4eb7-ae74-20456b229947-kube-api-access-l6nlg\" (UniqueName: \"kubernetes.io/projected/85b07127-3904-4eb7-ae74-20456b229947-kube-api-access-l6nlg\") pod \"85b07127-3904-4eb7-ae74-20456b229947\" (UID: \"85b07127-3904-4eb7-ae74-20456b229947\") " Nov 03 00:37:49.110998 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:49.110971 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85b07127-3904-4eb7-ae74-20456b229947-maas-api-openshift-ca" pod "85b07127-3904-4eb7-ae74-20456b229947" (UID: "85b07127-3904-4eb7-ae74-20456b229947"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:37:49.113341 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:49.113031 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85b07127-3904-4eb7-ae74-20456b229947-kube-api-access-l6nlg" pod "85b07127-3904-4eb7-ae74-20456b229947" (UID: "85b07127-3904-4eb7-ae74-20456b229947"). InnerVolumeSpecName "kube-api-access-l6nlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:37:49.113341 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:49.113320 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85b07127-3904-4eb7-ae74-20456b229947-maas-api-xks-ca" pod "85b07127-3904-4eb7-ae74-20456b229947" (UID: "85b07127-3904-4eb7-ae74-20456b229947"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:37:49.211627 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:49.211589 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85b07127-3904-4eb7-ae74-20456b229947-maas-api-xks-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:37:49.211627 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:49.211621 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85b07127-3904-4eb7-ae74-20456b229947-maas-api-openshift-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:37:49.211627 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:49.211632 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6nlg\" (UniqueName: \"kubernetes.io/projected/85b07127-3904-4eb7-ae74-20456b229947-kube-api-access-l6nlg\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:37:49.869636 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:49.869608 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4pdpj" Nov 03 00:37:49.869815 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:49.869645 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4pdpj" event={"ID":"85b07127-3904-4eb7-ae74-20456b229947","Type":"ContainerDied","Data":"68ca2d2ee1608c4e6e30db580913094cbe5056e075a7f9b57ce2aa3f65c84b80"} Nov 03 00:37:49.869815 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:37:49.869692 5076 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68ca2d2ee1608c4e6e30db580913094cbe5056e075a7f9b57ce2aa3f65c84b80" Nov 03 00:38:09.886359 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:38:09.886319 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:38:10.196570 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:38:10.196536 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6"] Nov 03 00:38:10.197076 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:38:10.197056 5076 cpu_manager.go:414] "containerMap: removing container" podUID="85b07127-3904-4eb7-ae74-20456b229947" containerName="revoke-keys" Nov 03 00:38:10.197181 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:38:10.197077 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85b07127-3904-4eb7-ae74-20456b229947" containerName="revoke-keys" Nov 03 00:38:10.197253 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:38:10.197243 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="85b07127-3904-4eb7-ae74-20456b229947" containerName="revoke-keys" Nov 03 00:38:10.200599 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:38:10.200576 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6" Nov 03 00:38:10.252040 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:38:10.252003 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-aa4c43-serving-cert\"" Nov 03 00:38:10.299262 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:38:10.299231 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-b7gfc\"" Nov 03 00:38:10.299438 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:38:10.299276 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 00:38:10.302014 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:38:10.301992 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6"] Nov 03 00:38:10.330247 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:38:10.330216 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/550b2a0c-8d7c-424c-9862-6a2c8fc8c555-metrics-certs\") pod \"maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6\" (UID: \"550b2a0c-8d7c-424c-9862-6a2c8fc8c555\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6" Nov 03 00:38:10.330403 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:38:10.330261 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppvqj\" (UniqueName: \"kubernetes.io/projected/550b2a0c-8d7c-424c-9862-6a2c8fc8c555-kube-api-access-ppvqj\") pod \"maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6\" (UID: \"550b2a0c-8d7c-424c-9862-6a2c8fc8c555\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6" Nov 03 00:38:10.330403 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:38:10.330377 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/550b2a0c-8d7c-424c-9862-6a2c8fc8c555-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6\" (UID: \"550b2a0c-8d7c-424c-9862-6a2c8fc8c555\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6" Nov 03 00:38:10.330491 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:38:10.330449 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/550b2a0c-8d7c-424c-9862-6a2c8fc8c555-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6\" (UID: \"550b2a0c-8d7c-424c-9862-6a2c8fc8c555\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6" Nov 03 00:38:10.431494 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:38:10.431458 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/550b2a0c-8d7c-424c-9862-6a2c8fc8c555-metrics-certs\") pod \"maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6\" (UID: \"550b2a0c-8d7c-424c-9862-6a2c8fc8c555\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6" Nov 03 00:38:10.431702 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:38:10.431506 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppvqj\" (UniqueName: \"kubernetes.io/projected/550b2a0c-8d7c-424c-9862-6a2c8fc8c555-kube-api-access-ppvqj\") pod \"maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6\" (UID: \"550b2a0c-8d7c-424c-9862-6a2c8fc8c555\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6" Nov 03 00:38:10.431702 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:38:10.431580 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/550b2a0c-8d7c-424c-9862-6a2c8fc8c555-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6\" (UID: \"550b2a0c-8d7c-424c-9862-6a2c8fc8c555\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6" Nov 03 00:38:10.431702 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:38:10.431658 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/550b2a0c-8d7c-424c-9862-6a2c8fc8c555-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6\" (UID: \"550b2a0c-8d7c-424c-9862-6a2c8fc8c555\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6" Nov 03 00:38:10.431867 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:38:10.431850 5076 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa4c43-serving-cert: secret "maas-api-e2e-worker-single-main-aa4c43-serving-cert" not found Nov 03 00:38:10.431945 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:38:10.431933 5076 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/550b2a0c-8d7c-424c-9862-6a2c8fc8c555-maas-api-tls podName:550b2a0c-8d7c-424c-9862-6a2c8fc8c555 nodeName:}" failed. No retries permitted until 2026-11-03 00:38:10.931912406 +0000 UTC m=+6339.900444188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/550b2a0c-8d7c-424c-9862-6a2c8fc8c555-maas-api-tls") pod "maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6" (UID: "550b2a0c-8d7c-424c-9862-6a2c8fc8c555") : secret "maas-api-e2e-worker-single-main-aa4c43-serving-cert" not found Nov 03 00:38:10.432308 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:38:10.432284 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/550b2a0c-8d7c-424c-9862-6a2c8fc8c555-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6\" (UID: \"550b2a0c-8d7c-424c-9862-6a2c8fc8c555\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6" Nov 03 00:38:10.433994 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:38:10.433972 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/550b2a0c-8d7c-424c-9862-6a2c8fc8c555-metrics-certs\") pod \"maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6\" (UID: \"550b2a0c-8d7c-424c-9862-6a2c8fc8c555\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6" Nov 03 00:38:10.486392 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:38:10.486312 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppvqj\" (UniqueName: \"kubernetes.io/projected/550b2a0c-8d7c-424c-9862-6a2c8fc8c555-kube-api-access-ppvqj\") pod \"maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6\" (UID: \"550b2a0c-8d7c-424c-9862-6a2c8fc8c555\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6" Nov 03 00:38:10.935802 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:38:10.935768 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/550b2a0c-8d7c-424c-9862-6a2c8fc8c555-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6\" (UID: \"550b2a0c-8d7c-424c-9862-6a2c8fc8c555\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6" Nov 03 00:38:10.938216 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:38:10.938188 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/550b2a0c-8d7c-424c-9862-6a2c8fc8c555-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6\" (UID: \"550b2a0c-8d7c-424c-9862-6a2c8fc8c555\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6" Nov 03 00:38:11.118556 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:38:11.118469 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6" Nov 03 00:38:11.494167 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:38:11.494136 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6"] Nov 03 00:38:11.495654 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:38:11.495628 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod550b2a0c_8d7c_424c_9862_6a2c8fc8c555.slice/crio-7d5d406cc58210acde5133324191e27601304f5ce0fc5e6a394741891216ed2e WatchSource:0}: Error finding container 7d5d406cc58210acde5133324191e27601304f5ce0fc5e6a394741891216ed2e: Status 404 returned error can't find the container with id 7d5d406cc58210acde5133324191e27601304f5ce0fc5e6a394741891216ed2e Nov 03 00:38:11.497047 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:38:11.497029 5076 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:38:11.999122 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:38:11.999085 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6" event={"ID":"550b2a0c-8d7c-424c-9862-6a2c8fc8c555","Type":"ContainerStarted","Data":"7d5d406cc58210acde5133324191e27601304f5ce0fc5e6a394741891216ed2e"} Nov 03 00:38:13.014234 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:38:13.014179 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6" event={"ID":"550b2a0c-8d7c-424c-9862-6a2c8fc8c555","Type":"ContainerStarted","Data":"bf576ee9546bb98e2626a9d5ce7a69f9a86f14ac461d21b4f88d87449ecb6b4e"} Nov 03 00:38:13.014736 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:38:13.014370 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6" Nov 03 00:38:13.043989 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:38:13.043944 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bbvrp"] Nov 03 00:38:13.091029 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:38:13.090991 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bbvrp"] Nov 03 00:38:13.147307 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:38:13.147270 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5737a98-d253-46ba-a47a-73c2d4a84985" path="/var/lib/kubelet/pods/e5737a98-d253-46ba-a47a-73c2d4a84985/volumes" Nov 03 00:38:13.149116 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:38:13.149071 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6" podStartSLOduration=2.213941472 podStartE2EDuration="3.149059359s" totalImagesPullingTime="935.117887ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:38:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:38:13.148080846 +0000 UTC m=+6342.116612631" watchObservedRunningTime="2026-11-03 00:38:13.149059359 +0000 UTC m=+6342.117591143" Nov 03 00:38:19.025834 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:38:19.025806 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6" Nov 03 00:38:39.044451 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:38:39.044416 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-25kwr"] Nov 03 00:38:39.091594 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:38:39.091540 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-25kwr"] Nov 03 00:38:39.149624 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:38:39.149585 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dca5053-8077-4cd0-936d-6ff45551dac5" path="/var/lib/kubelet/pods/1dca5053-8077-4cd0-936d-6ff45551dac5/volumes" Nov 03 00:38:39.883880 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:38:39.883844 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:38:51.050251 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:38:51.050212 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8tbdm"] Nov 03 00:38:51.097351 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:38:51.097310 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8tbdm"] Nov 03 00:38:51.146947 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:38:51.146909 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3202fb7b-a304-4b46-8663-57d35427b1ca" path="/var/lib/kubelet/pods/3202fb7b-a304-4b46-8663-57d35427b1ca/volumes" Nov 03 00:39:09.885099 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:39:09.885055 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:39:26.044416 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:39:26.044377 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xbb6j"] Nov 03 00:39:26.046650 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:39:26.046625 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xbb6j"] Nov 03 00:39:27.148111 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:39:27.148068 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="399e376c-5a9e-4d79-9f04-6c5bb148acbf" path="/var/lib/kubelet/pods/399e376c-5a9e-4d79-9f04-6c5bb148acbf/volumes" Nov 03 00:39:33.420685 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:39:33.420651 5076 scope.go:122] "RemoveContainer" containerID="fc4716fdc147f38ce68b3907ce4e1f2667cf9f8d9edb295c9bc45f1bb30d7aff" Nov 03 00:39:33.429711 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:39:33.429692 5076 scope.go:122] "RemoveContainer" containerID="a753bbf95f6932e0aa2e797169348957c0af0751c86a6e95aa9429b78e8d1ab6" Nov 03 00:39:37.046898 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:39:37.046850 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-67zfm"] Nov 03 00:39:37.094047 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:39:37.094012 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-67zfm"] Nov 03 00:39:37.146809 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:39:37.146775 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36b9297b-e093-4d79-980c-a208a4d51154" path="/var/lib/kubelet/pods/36b9297b-e093-4d79-980c-a208a4d51154/volumes" Nov 03 00:39:39.884085 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:39:39.884047 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:39:40.307995 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:39:40.307956 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6"] Nov 03 00:39:40.308207 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:39:40.308169 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6" podUID="550b2a0c-8d7c-424c-9862-6a2c8fc8c555" containerName="maas-api" containerID="cri-o://bf576ee9546bb98e2626a9d5ce7a69f9a86f14ac461d21b4f88d87449ecb6b4e" gracePeriod=30 Nov 03 00:39:41.551705 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:39:41.551673 5076 generic.go:356] "Generic (PLEG): container finished" podID="550b2a0c-8d7c-424c-9862-6a2c8fc8c555" containerID="bf576ee9546bb98e2626a9d5ce7a69f9a86f14ac461d21b4f88d87449ecb6b4e" exitCode=0 Nov 03 00:39:41.552125 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:39:41.551751 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6" event={"ID":"550b2a0c-8d7c-424c-9862-6a2c8fc8c555","Type":"ContainerDied","Data":"bf576ee9546bb98e2626a9d5ce7a69f9a86f14ac461d21b4f88d87449ecb6b4e"} Nov 03 00:39:41.552125 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:39:41.551805 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6" event={"ID":"550b2a0c-8d7c-424c-9862-6a2c8fc8c555","Type":"ContainerDied","Data":"7d5d406cc58210acde5133324191e27601304f5ce0fc5e6a394741891216ed2e"} Nov 03 00:39:41.552125 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:39:41.551827 5076 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d5d406cc58210acde5133324191e27601304f5ce0fc5e6a394741891216ed2e" Nov 03 00:39:41.569880 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:39:41.569851 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6" Nov 03 00:39:41.706884 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:39:41.706852 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/550b2a0c-8d7c-424c-9862-6a2c8fc8c555-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/550b2a0c-8d7c-424c-9862-6a2c8fc8c555-maas-api-tls\") pod \"550b2a0c-8d7c-424c-9862-6a2c8fc8c555\" (UID: \"550b2a0c-8d7c-424c-9862-6a2c8fc8c555\") " Nov 03 00:39:41.707140 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:39:41.706958 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/550b2a0c-8d7c-424c-9862-6a2c8fc8c555-metrics-certs\" (UniqueName: \"kubernetes.io/secret/550b2a0c-8d7c-424c-9862-6a2c8fc8c555-metrics-certs\") pod \"550b2a0c-8d7c-424c-9862-6a2c8fc8c555\" (UID: \"550b2a0c-8d7c-424c-9862-6a2c8fc8c555\") " Nov 03 00:39:41.707140 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:39:41.707007 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/550b2a0c-8d7c-424c-9862-6a2c8fc8c555-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/550b2a0c-8d7c-424c-9862-6a2c8fc8c555-openshift-service-ca\") pod \"550b2a0c-8d7c-424c-9862-6a2c8fc8c555\" (UID: \"550b2a0c-8d7c-424c-9862-6a2c8fc8c555\") " Nov 03 00:39:41.707140 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:39:41.707032 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/550b2a0c-8d7c-424c-9862-6a2c8fc8c555-kube-api-access-ppvqj\" (UniqueName: \"kubernetes.io/projected/550b2a0c-8d7c-424c-9862-6a2c8fc8c555-kube-api-access-ppvqj\") pod \"550b2a0c-8d7c-424c-9862-6a2c8fc8c555\" (UID: \"550b2a0c-8d7c-424c-9862-6a2c8fc8c555\") " Nov 03 00:39:41.707386 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:39:41.707353 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/550b2a0c-8d7c-424c-9862-6a2c8fc8c555-openshift-service-ca" pod "550b2a0c-8d7c-424c-9862-6a2c8fc8c555" (UID: "550b2a0c-8d7c-424c-9862-6a2c8fc8c555"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:39:41.709065 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:39:41.709036 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/550b2a0c-8d7c-424c-9862-6a2c8fc8c555-maas-api-tls" pod "550b2a0c-8d7c-424c-9862-6a2c8fc8c555" (UID: "550b2a0c-8d7c-424c-9862-6a2c8fc8c555"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:39:41.709188 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:39:41.709094 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/550b2a0c-8d7c-424c-9862-6a2c8fc8c555-metrics-certs" pod "550b2a0c-8d7c-424c-9862-6a2c8fc8c555" (UID: "550b2a0c-8d7c-424c-9862-6a2c8fc8c555"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:39:41.709508 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:39:41.709485 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/550b2a0c-8d7c-424c-9862-6a2c8fc8c555-kube-api-access-ppvqj" pod "550b2a0c-8d7c-424c-9862-6a2c8fc8c555" (UID: "550b2a0c-8d7c-424c-9862-6a2c8fc8c555"). InnerVolumeSpecName "kube-api-access-ppvqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:39:41.808279 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:39:41.808242 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/550b2a0c-8d7c-424c-9862-6a2c8fc8c555-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:39:41.808279 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:39:41.808274 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/550b2a0c-8d7c-424c-9862-6a2c8fc8c555-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:39:41.808279 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:39:41.808288 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/550b2a0c-8d7c-424c-9862-6a2c8fc8c555-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:39:41.808537 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:39:41.808297 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppvqj\" (UniqueName: \"kubernetes.io/projected/550b2a0c-8d7c-424c-9862-6a2c8fc8c555-kube-api-access-ppvqj\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:39:42.556640 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:39:42.556609 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6" Nov 03 00:39:42.717096 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:39:42.717060 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6"] Nov 03 00:39:42.763641 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:39:42.763596 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa4c43-7c59964ccc-nfzt6"] Nov 03 00:39:43.146957 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:39:43.146923 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="550b2a0c-8d7c-424c-9862-6a2c8fc8c555" path="/var/lib/kubelet/pods/550b2a0c-8d7c-424c-9862-6a2c8fc8c555/volumes" Nov 03 00:40:09.883695 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:40:09.883655 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:40:12.044185 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:12.044147 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d84vk"] Nov 03 00:40:12.094289 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:12.094248 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d84vk"] Nov 03 00:40:13.147420 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:13.147389 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5" path="/var/lib/kubelet/pods/ee4e4b4b-c5c8-4d09-9c02-da5bc3cdfcd5/volumes" Nov 03 00:40:14.118972 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:14.113157 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4pdpj"] Nov 03 00:40:14.159402 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:14.159367 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4pdpj"] Nov 03 00:40:14.292752 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:14.292718 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjtqd"] Nov 03 00:40:14.293243 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:14.293226 5076 cpu_manager.go:414] "containerMap: removing container" podUID="550b2a0c-8d7c-424c-9862-6a2c8fc8c555" containerName="maas-api" Nov 03 00:40:14.293306 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:14.293244 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="550b2a0c-8d7c-424c-9862-6a2c8fc8c555" containerName="maas-api" Nov 03 00:40:14.293345 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:14.293328 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="550b2a0c-8d7c-424c-9862-6a2c8fc8c555" containerName="maas-api" Nov 03 00:40:14.296937 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:14.296919 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjtqd" Nov 03 00:40:14.347088 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:14.347056 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:40:14.347963 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:14.347939 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:40:14.349913 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:14.349873 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjtqd"] Nov 03 00:40:14.438683 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:14.438652 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd30c2f2-b0ab-42e1-affa-c5ec030f8121-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-hjtqd\" (UID: \"fd30c2f2-b0ab-42e1-affa-c5ec030f8121\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjtqd" Nov 03 00:40:14.438846 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:14.438692 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfn8b\" (UniqueName: \"kubernetes.io/projected/fd30c2f2-b0ab-42e1-affa-c5ec030f8121-kube-api-access-lfn8b\") pod \"maas-api-revoke-sub-3d4156f1-hjtqd\" (UID: \"fd30c2f2-b0ab-42e1-affa-c5ec030f8121\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjtqd" Nov 03 00:40:14.438846 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:14.438775 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd30c2f2-b0ab-42e1-affa-c5ec030f8121-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-hjtqd\" (UID: \"fd30c2f2-b0ab-42e1-affa-c5ec030f8121\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjtqd" Nov 03 00:40:14.539744 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:14.539696 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd30c2f2-b0ab-42e1-affa-c5ec030f8121-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-hjtqd\" (UID: \"fd30c2f2-b0ab-42e1-affa-c5ec030f8121\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjtqd" Nov 03 00:40:14.539744 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:14.539746 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfn8b\" (UniqueName: \"kubernetes.io/projected/fd30c2f2-b0ab-42e1-affa-c5ec030f8121-kube-api-access-lfn8b\") pod \"maas-api-revoke-sub-3d4156f1-hjtqd\" (UID: \"fd30c2f2-b0ab-42e1-affa-c5ec030f8121\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjtqd" Nov 03 00:40:14.540072 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:14.539942 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd30c2f2-b0ab-42e1-affa-c5ec030f8121-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-hjtqd\" (UID: \"fd30c2f2-b0ab-42e1-affa-c5ec030f8121\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjtqd" Nov 03 00:40:14.540409 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:14.540380 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd30c2f2-b0ab-42e1-affa-c5ec030f8121-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-hjtqd\" (UID: \"fd30c2f2-b0ab-42e1-affa-c5ec030f8121\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjtqd" Nov 03 00:40:14.542209 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:14.542191 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd30c2f2-b0ab-42e1-affa-c5ec030f8121-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-hjtqd\" (UID: \"fd30c2f2-b0ab-42e1-affa-c5ec030f8121\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjtqd" Nov 03 00:40:14.594051 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:14.594015 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfn8b\" (UniqueName: \"kubernetes.io/projected/fd30c2f2-b0ab-42e1-affa-c5ec030f8121-kube-api-access-lfn8b\") pod \"maas-api-revoke-sub-3d4156f1-hjtqd\" (UID: \"fd30c2f2-b0ab-42e1-affa-c5ec030f8121\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjtqd" Nov 03 00:40:14.610034 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:14.610005 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjtqd" Nov 03 00:40:14.979196 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:14.979166 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjtqd"] Nov 03 00:40:14.980676 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:40:14.980650 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd30c2f2_b0ab_42e1_affa_c5ec030f8121.slice/crio-4c6004d927ae817beb7d4816f91545a5d06e9553b511eb8990453bfd98fd4065 WatchSource:0}: Error finding container 4c6004d927ae817beb7d4816f91545a5d06e9553b511eb8990453bfd98fd4065: Status 404 returned error can't find the container with id 4c6004d927ae817beb7d4816f91545a5d06e9553b511eb8990453bfd98fd4065 Nov 03 00:40:15.148232 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:15.148200 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85b07127-3904-4eb7-ae74-20456b229947" path="/var/lib/kubelet/pods/85b07127-3904-4eb7-ae74-20456b229947/volumes" Nov 03 00:40:15.737560 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:15.737527 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjtqd" event={"ID":"fd30c2f2-b0ab-42e1-affa-c5ec030f8121","Type":"ContainerStarted","Data":"d85eba31a9eda17d2a54ab259d2446f66e7792593f048778831c2e212a844a60"} Nov 03 00:40:15.737981 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:15.737576 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjtqd" event={"ID":"fd30c2f2-b0ab-42e1-affa-c5ec030f8121","Type":"ContainerStarted","Data":"4c6004d927ae817beb7d4816f91545a5d06e9553b511eb8990453bfd98fd4065"} Nov 03 00:40:15.848020 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:15.847961 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjtqd" podStartSLOduration=1.84794635 podStartE2EDuration="1.84794635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:40:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:15.847312417 +0000 UTC m=+6464.815844201" watchObservedRunningTime="2026-11-03 00:40:15.84794635 +0000 UTC m=+6464.816478169" Nov 03 00:40:16.744580 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:16.744543 5076 generic.go:356] "Generic (PLEG): container finished" podID="fd30c2f2-b0ab-42e1-affa-c5ec030f8121" containerID="d85eba31a9eda17d2a54ab259d2446f66e7792593f048778831c2e212a844a60" exitCode=0 Nov 03 00:40:16.744997 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:16.744598 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjtqd" event={"ID":"fd30c2f2-b0ab-42e1-affa-c5ec030f8121","Type":"ContainerDied","Data":"d85eba31a9eda17d2a54ab259d2446f66e7792593f048778831c2e212a844a60"} Nov 03 00:40:17.897620 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:17.897596 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjtqd" Nov 03 00:40:17.973536 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:17.973508 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd30c2f2-b0ab-42e1-affa-c5ec030f8121-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd30c2f2-b0ab-42e1-affa-c5ec030f8121-maas-api-openshift-ca\") pod \"fd30c2f2-b0ab-42e1-affa-c5ec030f8121\" (UID: \"fd30c2f2-b0ab-42e1-affa-c5ec030f8121\") " Nov 03 00:40:17.973718 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:17.973587 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd30c2f2-b0ab-42e1-affa-c5ec030f8121-kube-api-access-lfn8b\" (UniqueName: \"kubernetes.io/projected/fd30c2f2-b0ab-42e1-affa-c5ec030f8121-kube-api-access-lfn8b\") pod \"fd30c2f2-b0ab-42e1-affa-c5ec030f8121\" (UID: \"fd30c2f2-b0ab-42e1-affa-c5ec030f8121\") " Nov 03 00:40:17.973718 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:17.973684 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd30c2f2-b0ab-42e1-affa-c5ec030f8121-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd30c2f2-b0ab-42e1-affa-c5ec030f8121-maas-api-xks-ca\") pod \"fd30c2f2-b0ab-42e1-affa-c5ec030f8121\" (UID: \"fd30c2f2-b0ab-42e1-affa-c5ec030f8121\") " Nov 03 00:40:17.973970 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:17.973946 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd30c2f2-b0ab-42e1-affa-c5ec030f8121-maas-api-openshift-ca" pod "fd30c2f2-b0ab-42e1-affa-c5ec030f8121" (UID: "fd30c2f2-b0ab-42e1-affa-c5ec030f8121"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:40:17.974115 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:17.974095 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd30c2f2-b0ab-42e1-affa-c5ec030f8121-maas-api-openshift-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:40:17.975703 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:17.975672 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd30c2f2-b0ab-42e1-affa-c5ec030f8121-kube-api-access-lfn8b" pod "fd30c2f2-b0ab-42e1-affa-c5ec030f8121" (UID: "fd30c2f2-b0ab-42e1-affa-c5ec030f8121"). InnerVolumeSpecName "kube-api-access-lfn8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:40:17.975793 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:17.975707 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd30c2f2-b0ab-42e1-affa-c5ec030f8121-maas-api-xks-ca" pod "fd30c2f2-b0ab-42e1-affa-c5ec030f8121" (UID: "fd30c2f2-b0ab-42e1-affa-c5ec030f8121"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:40:18.075108 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:18.075031 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfn8b\" (UniqueName: \"kubernetes.io/projected/fd30c2f2-b0ab-42e1-affa-c5ec030f8121-kube-api-access-lfn8b\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:40:18.075108 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:18.075061 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd30c2f2-b0ab-42e1-affa-c5ec030f8121-maas-api-xks-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:40:18.757623 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:18.757588 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjtqd" event={"ID":"fd30c2f2-b0ab-42e1-affa-c5ec030f8121","Type":"ContainerDied","Data":"4c6004d927ae817beb7d4816f91545a5d06e9553b511eb8990453bfd98fd4065"} Nov 03 00:40:18.757623 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:18.757614 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjtqd" Nov 03 00:40:18.757851 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:18.757663 5076 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c6004d927ae817beb7d4816f91545a5d06e9553b511eb8990453bfd98fd4065" Nov 03 00:40:33.051365 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:33.051327 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9zmbh"] Nov 03 00:40:33.098468 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:33.098429 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9zmbh"] Nov 03 00:40:33.146731 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:33.146695 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50097625-1a31-4ab8-9d17-8901ebb130f8" path="/var/lib/kubelet/pods/50097625-1a31-4ab8-9d17-8901ebb130f8/volumes" Nov 03 00:40:33.459362 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:33.459327 5076 scope.go:122] "RemoveContainer" containerID="221685e77f34d3b46086d18eaf489569689b877bac3e34e32b262096631aa284" Nov 03 00:40:33.470759 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:33.470733 5076 scope.go:122] "RemoveContainer" containerID="eb8a468e570ea9e774305dcd3440d0aa922e8e6a9ed3d92abbcf5e550fb6baf3" Nov 03 00:40:33.479400 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:33.479376 5076 scope.go:122] "RemoveContainer" containerID="69a72a6e60b45e6b84b8731c66d60c99ce7d475bfd8610dae4daf4f479d6f95f" Nov 03 00:40:39.884117 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:40:39.884068 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:40:55.052113 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:55.052077 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p8t9x"] Nov 03 00:40:55.099054 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:55.099023 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p8t9x"] Nov 03 00:40:55.147027 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:40:55.146991 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="168923e9-9595-4a47-a0a5-1aba6d6c919c" path="/var/lib/kubelet/pods/168923e9-9595-4a47-a0a5-1aba6d6c919c/volumes" Nov 03 00:41:09.883209 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:41:09.883167 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:41:13.044692 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:41:13.044609 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-kjvs7"] Nov 03 00:41:13.091943 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:41:13.091910 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-kjvs7"] Nov 03 00:41:13.147882 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:41:13.147852 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5651ce1a-b668-48ff-95d2-f80078eb7863" path="/var/lib/kubelet/pods/5651ce1a-b668-48ff-95d2-f80078eb7863/volumes" Nov 03 00:41:33.513184 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:41:33.513151 5076 scope.go:122] "RemoveContainer" containerID="844c31e1f9c495fe1257f7e1326c1585727f396bb7340d55c23fc602b3311a6f" Nov 03 00:41:33.523769 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:41:33.523742 5076 scope.go:122] "RemoveContainer" containerID="6c9c6c51cd53462ad56d06d481de3b01c1c5a8b080b1f93e1b5c4be8ef0c201c" Nov 03 00:41:33.533306 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:41:33.533283 5076 scope.go:122] "RemoveContainer" containerID="4bcd429f27383747ba395f569b0784c72192d7419a97e60711336dd7d45f0d83" Nov 03 00:41:39.886083 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:41:39.886042 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:42:01.474914 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:01.474023 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gctbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:42:01.474914 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:01.474126 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" podUID="d6db6fe8-b76c-4878-b0f3-bd347b2d0cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:42:01.819136 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:01.819023 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-784vq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:42:01.819440 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:01.819390 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" podUID="506655af-c50a-4207-906e-9f705fc8f9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:42:03.241688 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:03.241646 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-m9q7q"] Nov 03 00:42:03.242134 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:03.242119 5076 cpu_manager.go:414] "containerMap: removing container" podUID="fd30c2f2-b0ab-42e1-affa-c5ec030f8121" containerName="revoke-keys" Nov 03 00:42:03.242186 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:03.242134 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd30c2f2-b0ab-42e1-affa-c5ec030f8121" containerName="revoke-keys" Nov 03 00:42:03.242226 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:03.242202 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd30c2f2-b0ab-42e1-affa-c5ec030f8121" containerName="revoke-keys" Nov 03 00:42:03.245334 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:03.245318 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-m9q7q" Nov 03 00:42:03.270016 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:03.269989 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9n6k\" (UniqueName: \"kubernetes.io/projected/fd7427ec-e02e-4e23-ae32-27ae6ddfbad4-kube-api-access-s9n6k\") pod \"maas-api-revoke-sub-d576e68b-m9q7q\" (UID: \"fd7427ec-e02e-4e23-ae32-27ae6ddfbad4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-m9q7q" Nov 03 00:42:03.270138 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:03.270038 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd7427ec-e02e-4e23-ae32-27ae6ddfbad4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-m9q7q\" (UID: \"fd7427ec-e02e-4e23-ae32-27ae6ddfbad4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-m9q7q" Nov 03 00:42:03.270177 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:03.270152 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd7427ec-e02e-4e23-ae32-27ae6ddfbad4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-m9q7q\" (UID: \"fd7427ec-e02e-4e23-ae32-27ae6ddfbad4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-m9q7q" Nov 03 00:42:03.297942 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:03.297912 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-m9q7q"] Nov 03 00:42:03.342863 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:03.342828 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:42:03.342863 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:03.342865 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:42:03.371455 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:03.371426 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9n6k\" (UniqueName: \"kubernetes.io/projected/fd7427ec-e02e-4e23-ae32-27ae6ddfbad4-kube-api-access-s9n6k\") pod \"maas-api-revoke-sub-d576e68b-m9q7q\" (UID: \"fd7427ec-e02e-4e23-ae32-27ae6ddfbad4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-m9q7q" Nov 03 00:42:03.371645 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:03.371468 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd7427ec-e02e-4e23-ae32-27ae6ddfbad4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-m9q7q\" (UID: \"fd7427ec-e02e-4e23-ae32-27ae6ddfbad4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-m9q7q" Nov 03 00:42:03.371716 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:03.371657 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd7427ec-e02e-4e23-ae32-27ae6ddfbad4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-m9q7q\" (UID: \"fd7427ec-e02e-4e23-ae32-27ae6ddfbad4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-m9q7q" Nov 03 00:42:03.372300 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:03.372278 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd7427ec-e02e-4e23-ae32-27ae6ddfbad4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-m9q7q\" (UID: \"fd7427ec-e02e-4e23-ae32-27ae6ddfbad4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-m9q7q" Nov 03 00:42:03.373802 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:03.373782 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd7427ec-e02e-4e23-ae32-27ae6ddfbad4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-m9q7q\" (UID: \"fd7427ec-e02e-4e23-ae32-27ae6ddfbad4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-m9q7q" Nov 03 00:42:03.425578 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:03.425545 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9n6k\" (UniqueName: \"kubernetes.io/projected/fd7427ec-e02e-4e23-ae32-27ae6ddfbad4-kube-api-access-s9n6k\") pod \"maas-api-revoke-sub-d576e68b-m9q7q\" (UID: \"fd7427ec-e02e-4e23-ae32-27ae6ddfbad4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-m9q7q" Nov 03 00:42:03.558629 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:03.558546 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-m9q7q" Nov 03 00:42:03.923201 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:03.923173 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-m9q7q"] Nov 03 00:42:03.924173 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:42:03.924144 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd7427ec_e02e_4e23_ae32_27ae6ddfbad4.slice/crio-e00e6cf3bd16ee2d9b9ed5f994c1b951867d61e1dd33c02d567da2733a6aa96e WatchSource:0}: Error finding container e00e6cf3bd16ee2d9b9ed5f994c1b951867d61e1dd33c02d567da2733a6aa96e: Status 404 returned error can't find the container with id e00e6cf3bd16ee2d9b9ed5f994c1b951867d61e1dd33c02d567da2733a6aa96e Nov 03 00:42:04.367218 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:04.367177 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-m9q7q" event={"ID":"fd7427ec-e02e-4e23-ae32-27ae6ddfbad4","Type":"ContainerStarted","Data":"041980d4ec32200b8c503b1b34f1cae8aecc3199d25a6d5b8f7dc92b1e861662"} Nov 03 00:42:04.367718 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:04.367235 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-m9q7q" event={"ID":"fd7427ec-e02e-4e23-ae32-27ae6ddfbad4","Type":"ContainerStarted","Data":"e00e6cf3bd16ee2d9b9ed5f994c1b951867d61e1dd33c02d567da2733a6aa96e"} Nov 03 00:42:04.476870 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:04.476768 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-m9q7q" podStartSLOduration=1.476753887 podStartE2EDuration="1.476753887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:42:04.475062901 +0000 UTC m=+6573.443594684" watchObservedRunningTime="2026-11-03 00:42:04.476753887 +0000 UTC m=+6573.445285671" Nov 03 00:42:05.374078 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:05.374045 5076 generic.go:356] "Generic (PLEG): container finished" podID="fd7427ec-e02e-4e23-ae32-27ae6ddfbad4" containerID="041980d4ec32200b8c503b1b34f1cae8aecc3199d25a6d5b8f7dc92b1e861662" exitCode=0 Nov 03 00:42:05.374472 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:05.374133 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-m9q7q" event={"ID":"fd7427ec-e02e-4e23-ae32-27ae6ddfbad4","Type":"ContainerDied","Data":"041980d4ec32200b8c503b1b34f1cae8aecc3199d25a6d5b8f7dc92b1e861662"} Nov 03 00:42:06.518585 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:06.518558 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-m9q7q" Nov 03 00:42:06.607979 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:06.607943 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd7427ec-e02e-4e23-ae32-27ae6ddfbad4-kube-api-access-s9n6k\" (UniqueName: \"kubernetes.io/projected/fd7427ec-e02e-4e23-ae32-27ae6ddfbad4-kube-api-access-s9n6k\") pod \"fd7427ec-e02e-4e23-ae32-27ae6ddfbad4\" (UID: \"fd7427ec-e02e-4e23-ae32-27ae6ddfbad4\") " Nov 03 00:42:06.608154 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:06.608078 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd7427ec-e02e-4e23-ae32-27ae6ddfbad4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd7427ec-e02e-4e23-ae32-27ae6ddfbad4-maas-api-openshift-ca\") pod \"fd7427ec-e02e-4e23-ae32-27ae6ddfbad4\" (UID: \"fd7427ec-e02e-4e23-ae32-27ae6ddfbad4\") " Nov 03 00:42:06.608154 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:06.608145 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd7427ec-e02e-4e23-ae32-27ae6ddfbad4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd7427ec-e02e-4e23-ae32-27ae6ddfbad4-maas-api-xks-ca\") pod \"fd7427ec-e02e-4e23-ae32-27ae6ddfbad4\" (UID: \"fd7427ec-e02e-4e23-ae32-27ae6ddfbad4\") " Nov 03 00:42:06.608436 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:06.608411 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd7427ec-e02e-4e23-ae32-27ae6ddfbad4-maas-api-openshift-ca" pod "fd7427ec-e02e-4e23-ae32-27ae6ddfbad4" (UID: "fd7427ec-e02e-4e23-ae32-27ae6ddfbad4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:42:06.610086 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:06.610059 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd7427ec-e02e-4e23-ae32-27ae6ddfbad4-kube-api-access-s9n6k" pod "fd7427ec-e02e-4e23-ae32-27ae6ddfbad4" (UID: "fd7427ec-e02e-4e23-ae32-27ae6ddfbad4"). InnerVolumeSpecName "kube-api-access-s9n6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:42:06.610194 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:06.610147 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd7427ec-e02e-4e23-ae32-27ae6ddfbad4-maas-api-xks-ca" pod "fd7427ec-e02e-4e23-ae32-27ae6ddfbad4" (UID: "fd7427ec-e02e-4e23-ae32-27ae6ddfbad4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:42:06.709904 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:06.709850 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd7427ec-e02e-4e23-ae32-27ae6ddfbad4-maas-api-xks-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:42:06.709904 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:06.709883 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9n6k\" (UniqueName: \"kubernetes.io/projected/fd7427ec-e02e-4e23-ae32-27ae6ddfbad4-kube-api-access-s9n6k\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:42:06.710130 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:06.709922 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd7427ec-e02e-4e23-ae32-27ae6ddfbad4-maas-api-openshift-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:42:07.386181 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:07.386151 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-m9q7q" Nov 03 00:42:07.386400 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:07.386187 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-m9q7q" event={"ID":"fd7427ec-e02e-4e23-ae32-27ae6ddfbad4","Type":"ContainerDied","Data":"e00e6cf3bd16ee2d9b9ed5f994c1b951867d61e1dd33c02d567da2733a6aa96e"} Nov 03 00:42:07.386400 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:07.386235 5076 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e00e6cf3bd16ee2d9b9ed5f994c1b951867d61e1dd33c02d567da2733a6aa96e" Nov 03 00:42:09.882505 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:42:09.882466 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:42:20.043879 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:20.043835 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7cfh8"] Nov 03 00:42:20.091011 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:20.090973 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7cfh8"] Nov 03 00:42:21.151375 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:21.151333 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="455a916c-ca5a-4495-b27a-7d51d7a64118" path="/var/lib/kubelet/pods/455a916c-ca5a-4495-b27a-7d51d7a64118/volumes" Nov 03 00:42:24.712749 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:24.712717 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4vvkh"] Nov 03 00:42:24.713208 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:24.713191 5076 cpu_manager.go:414] "containerMap: removing container" podUID="fd7427ec-e02e-4e23-ae32-27ae6ddfbad4" containerName="revoke-keys" Nov 03 00:42:24.713257 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:24.713209 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd7427ec-e02e-4e23-ae32-27ae6ddfbad4" containerName="revoke-keys" Nov 03 00:42:24.713323 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:24.713312 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd7427ec-e02e-4e23-ae32-27ae6ddfbad4" containerName="revoke-keys" Nov 03 00:42:24.716511 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:24.716495 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4vvkh" Nov 03 00:42:24.769727 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:24.769694 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:42:24.815177 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:24.815145 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/578177a7-bb81-45fc-a26c-d99a99def956-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4vvkh\" (UID: \"578177a7-bb81-45fc-a26c-d99a99def956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4vvkh" Nov 03 00:42:24.815347 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:24.815282 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/578177a7-bb81-45fc-a26c-d99a99def956-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4vvkh\" (UID: \"578177a7-bb81-45fc-a26c-d99a99def956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4vvkh" Nov 03 00:42:24.815347 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:24.815337 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh6wx\" (UniqueName: \"kubernetes.io/projected/578177a7-bb81-45fc-a26c-d99a99def956-kube-api-access-xh6wx\") pod \"maas-api-revoke-sub-d2bc93bc-4vvkh\" (UID: \"578177a7-bb81-45fc-a26c-d99a99def956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4vvkh" Nov 03 00:42:24.817062 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:24.817040 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:42:24.819430 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:24.819405 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4vvkh"] Nov 03 00:42:24.916429 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:24.916393 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/578177a7-bb81-45fc-a26c-d99a99def956-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4vvkh\" (UID: \"578177a7-bb81-45fc-a26c-d99a99def956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4vvkh" Nov 03 00:42:24.916607 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:24.916513 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/578177a7-bb81-45fc-a26c-d99a99def956-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4vvkh\" (UID: \"578177a7-bb81-45fc-a26c-d99a99def956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4vvkh" Nov 03 00:42:24.916607 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:24.916553 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh6wx\" (UniqueName: \"kubernetes.io/projected/578177a7-bb81-45fc-a26c-d99a99def956-kube-api-access-xh6wx\") pod \"maas-api-revoke-sub-d2bc93bc-4vvkh\" (UID: \"578177a7-bb81-45fc-a26c-d99a99def956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4vvkh" Nov 03 00:42:24.917395 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:24.917365 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/578177a7-bb81-45fc-a26c-d99a99def956-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4vvkh\" (UID: \"578177a7-bb81-45fc-a26c-d99a99def956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4vvkh" Nov 03 00:42:24.918810 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:24.918785 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/578177a7-bb81-45fc-a26c-d99a99def956-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4vvkh\" (UID: \"578177a7-bb81-45fc-a26c-d99a99def956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4vvkh" Nov 03 00:42:24.970996 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:24.970921 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh6wx\" (UniqueName: \"kubernetes.io/projected/578177a7-bb81-45fc-a26c-d99a99def956-kube-api-access-xh6wx\") pod \"maas-api-revoke-sub-d2bc93bc-4vvkh\" (UID: \"578177a7-bb81-45fc-a26c-d99a99def956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4vvkh" Nov 03 00:42:25.029676 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:25.029647 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4vvkh" Nov 03 00:42:25.394657 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:25.394630 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4vvkh"] Nov 03 00:42:25.396654 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:42:25.396622 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod578177a7_bb81_45fc_a26c_d99a99def956.slice/crio-d58942c9cae9b94304e951c6f67ae476c5a2b936a05a8772db1080baa739fec5 WatchSource:0}: Error finding container d58942c9cae9b94304e951c6f67ae476c5a2b936a05a8772db1080baa739fec5: Status 404 returned error can't find the container with id d58942c9cae9b94304e951c6f67ae476c5a2b936a05a8772db1080baa739fec5 Nov 03 00:42:25.513371 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:25.513334 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4vvkh" event={"ID":"578177a7-bb81-45fc-a26c-d99a99def956","Type":"ContainerStarted","Data":"d58942c9cae9b94304e951c6f67ae476c5a2b936a05a8772db1080baa739fec5"} Nov 03 00:42:26.520423 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:26.520388 5076 generic.go:356] "Generic (PLEG): container finished" podID="578177a7-bb81-45fc-a26c-d99a99def956" containerID="bb00b7ca054f78639cd56bf5174bbcf92d6bae6a4a811c5b41da111afa13173f" exitCode=0 Nov 03 00:42:26.520824 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:26.520474 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4vvkh" event={"ID":"578177a7-bb81-45fc-a26c-d99a99def956","Type":"ContainerDied","Data":"bb00b7ca054f78639cd56bf5174bbcf92d6bae6a4a811c5b41da111afa13173f"} Nov 03 00:42:27.671264 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:27.671240 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4vvkh" Nov 03 00:42:27.740655 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:27.740619 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/578177a7-bb81-45fc-a26c-d99a99def956-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/578177a7-bb81-45fc-a26c-d99a99def956-maas-api-xks-ca\") pod \"578177a7-bb81-45fc-a26c-d99a99def956\" (UID: \"578177a7-bb81-45fc-a26c-d99a99def956\") " Nov 03 00:42:27.740856 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:27.740708 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/578177a7-bb81-45fc-a26c-d99a99def956-kube-api-access-xh6wx\" (UniqueName: \"kubernetes.io/projected/578177a7-bb81-45fc-a26c-d99a99def956-kube-api-access-xh6wx\") pod \"578177a7-bb81-45fc-a26c-d99a99def956\" (UID: \"578177a7-bb81-45fc-a26c-d99a99def956\") " Nov 03 00:42:27.740856 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:27.740812 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/578177a7-bb81-45fc-a26c-d99a99def956-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/578177a7-bb81-45fc-a26c-d99a99def956-maas-api-openshift-ca\") pod \"578177a7-bb81-45fc-a26c-d99a99def956\" (UID: \"578177a7-bb81-45fc-a26c-d99a99def956\") " Nov 03 00:42:27.741151 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:27.741130 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/578177a7-bb81-45fc-a26c-d99a99def956-maas-api-openshift-ca" pod "578177a7-bb81-45fc-a26c-d99a99def956" (UID: "578177a7-bb81-45fc-a26c-d99a99def956"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:42:27.741249 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:27.741237 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/578177a7-bb81-45fc-a26c-d99a99def956-maas-api-openshift-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:42:27.742707 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:27.742683 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/578177a7-bb81-45fc-a26c-d99a99def956-maas-api-xks-ca" pod "578177a7-bb81-45fc-a26c-d99a99def956" (UID: "578177a7-bb81-45fc-a26c-d99a99def956"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:42:27.742797 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:27.742755 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/578177a7-bb81-45fc-a26c-d99a99def956-kube-api-access-xh6wx" pod "578177a7-bb81-45fc-a26c-d99a99def956" (UID: "578177a7-bb81-45fc-a26c-d99a99def956"). InnerVolumeSpecName "kube-api-access-xh6wx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:42:27.842498 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:27.842418 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh6wx\" (UniqueName: \"kubernetes.io/projected/578177a7-bb81-45fc-a26c-d99a99def956-kube-api-access-xh6wx\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:42:27.842498 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:27.842447 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/578177a7-bb81-45fc-a26c-d99a99def956-maas-api-xks-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:42:28.537004 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:28.536970 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4vvkh" Nov 03 00:42:28.537222 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:28.537010 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4vvkh" event={"ID":"578177a7-bb81-45fc-a26c-d99a99def956","Type":"ContainerDied","Data":"d58942c9cae9b94304e951c6f67ae476c5a2b936a05a8772db1080baa739fec5"} Nov 03 00:42:28.537222 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:28.537059 5076 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d58942c9cae9b94304e951c6f67ae476c5a2b936a05a8772db1080baa739fec5" Nov 03 00:42:31.046956 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:31.046918 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-f2ddv"] Nov 03 00:42:31.094354 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:31.094321 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-f2ddv"] Nov 03 00:42:31.147976 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:31.147946 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cc0827f-b2ba-4586-bcc4-d07ded683156" path="/var/lib/kubelet/pods/6cc0827f-b2ba-4586-bcc4-d07ded683156/volumes" Nov 03 00:42:32.365066 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:32.365038 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 03 00:42:32.458909 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:32.458870 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 03 00:42:33.563610 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:33.563578 5076 scope.go:122] "RemoveContainer" containerID="d8afdfb3f2c610d307391ef6772ab086dfcef37327e1d3686bd2a269ec670313" Nov 03 00:42:33.572777 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:33.572755 5076 scope.go:122] "RemoveContainer" containerID="24dbc5940e7c01cbd41b32d684b1712bb3671dbda811e7f9794c5289cacb5cb5" Nov 03 00:42:33.581427 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:33.581408 5076 scope.go:122] "RemoveContainer" containerID="0293cbe20df172493e03e2610915e402d4e41adf31cf854176f69fba05bec9d3" Nov 03 00:42:37.730837 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:37.730802 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-65f5668669-x5rsr"] Nov 03 00:42:37.731359 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:37.731272 5076 cpu_manager.go:414] "containerMap: removing container" podUID="578177a7-bb81-45fc-a26c-d99a99def956" containerName="revoke-keys" Nov 03 00:42:37.731359 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:37.731284 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="578177a7-bb81-45fc-a26c-d99a99def956" containerName="revoke-keys" Nov 03 00:42:37.731359 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:37.731357 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="578177a7-bb81-45fc-a26c-d99a99def956" containerName="revoke-keys" Nov 03 00:42:37.736134 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:37.736116 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65f5668669-x5rsr" Nov 03 00:42:37.788683 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:37.788653 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-65f5668669-x5rsr"] Nov 03 00:42:37.833546 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:37.833513 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-b7gfc\"" Nov 03 00:42:37.833546 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:37.833513 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 00:42:37.833546 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:37.833525 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 00:42:37.852586 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:37.852549 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm6bx\" (UniqueName: \"kubernetes.io/projected/8089298b-3fd9-4bea-a592-525f4a65afdd-kube-api-access-cm6bx\") pod \"maas-api-65f5668669-x5rsr\" (UID: \"8089298b-3fd9-4bea-a592-525f4a65afdd\") " pod="odh-ai-gateway-infra/maas-api-65f5668669-x5rsr" Nov 03 00:42:37.852756 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:37.852707 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8089298b-3fd9-4bea-a592-525f4a65afdd-metrics-certs\") pod \"maas-api-65f5668669-x5rsr\" (UID: \"8089298b-3fd9-4bea-a592-525f4a65afdd\") " pod="odh-ai-gateway-infra/maas-api-65f5668669-x5rsr" Nov 03 00:42:37.852813 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:37.852756 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8089298b-3fd9-4bea-a592-525f4a65afdd-openshift-service-ca\") pod \"maas-api-65f5668669-x5rsr\" (UID: \"8089298b-3fd9-4bea-a592-525f4a65afdd\") " pod="odh-ai-gateway-infra/maas-api-65f5668669-x5rsr" Nov 03 00:42:37.852922 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:37.852880 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8089298b-3fd9-4bea-a592-525f4a65afdd-maas-api-tls\") pod \"maas-api-65f5668669-x5rsr\" (UID: \"8089298b-3fd9-4bea-a592-525f4a65afdd\") " pod="odh-ai-gateway-infra/maas-api-65f5668669-x5rsr" Nov 03 00:42:37.954198 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:37.954166 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8089298b-3fd9-4bea-a592-525f4a65afdd-metrics-certs\") pod \"maas-api-65f5668669-x5rsr\" (UID: \"8089298b-3fd9-4bea-a592-525f4a65afdd\") " pod="odh-ai-gateway-infra/maas-api-65f5668669-x5rsr" Nov 03 00:42:37.954393 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:37.954215 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8089298b-3fd9-4bea-a592-525f4a65afdd-openshift-service-ca\") pod \"maas-api-65f5668669-x5rsr\" (UID: \"8089298b-3fd9-4bea-a592-525f4a65afdd\") " pod="odh-ai-gateway-infra/maas-api-65f5668669-x5rsr" Nov 03 00:42:37.954393 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:37.954271 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8089298b-3fd9-4bea-a592-525f4a65afdd-maas-api-tls\") pod \"maas-api-65f5668669-x5rsr\" (UID: \"8089298b-3fd9-4bea-a592-525f4a65afdd\") " pod="odh-ai-gateway-infra/maas-api-65f5668669-x5rsr" Nov 03 00:42:37.954393 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:37.954308 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm6bx\" (UniqueName: \"kubernetes.io/projected/8089298b-3fd9-4bea-a592-525f4a65afdd-kube-api-access-cm6bx\") pod \"maas-api-65f5668669-x5rsr\" (UID: \"8089298b-3fd9-4bea-a592-525f4a65afdd\") " pod="odh-ai-gateway-infra/maas-api-65f5668669-x5rsr" Nov 03 00:42:37.954984 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:37.954956 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8089298b-3fd9-4bea-a592-525f4a65afdd-openshift-service-ca\") pod \"maas-api-65f5668669-x5rsr\" (UID: \"8089298b-3fd9-4bea-a592-525f4a65afdd\") " pod="odh-ai-gateway-infra/maas-api-65f5668669-x5rsr" Nov 03 00:42:37.956722 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:37.956698 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8089298b-3fd9-4bea-a592-525f4a65afdd-maas-api-tls\") pod \"maas-api-65f5668669-x5rsr\" (UID: \"8089298b-3fd9-4bea-a592-525f4a65afdd\") " pod="odh-ai-gateway-infra/maas-api-65f5668669-x5rsr" Nov 03 00:42:37.956807 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:37.956747 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8089298b-3fd9-4bea-a592-525f4a65afdd-metrics-certs\") pod \"maas-api-65f5668669-x5rsr\" (UID: \"8089298b-3fd9-4bea-a592-525f4a65afdd\") " pod="odh-ai-gateway-infra/maas-api-65f5668669-x5rsr" Nov 03 00:42:38.008324 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:38.008252 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm6bx\" (UniqueName: \"kubernetes.io/projected/8089298b-3fd9-4bea-a592-525f4a65afdd-kube-api-access-cm6bx\") pod \"maas-api-65f5668669-x5rsr\" (UID: \"8089298b-3fd9-4bea-a592-525f4a65afdd\") " pod="odh-ai-gateway-infra/maas-api-65f5668669-x5rsr" Nov 03 00:42:38.049125 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:38.049077 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65f5668669-x5rsr" Nov 03 00:42:38.417131 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:38.417106 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-65f5668669-x5rsr"] Nov 03 00:42:38.418185 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:42:38.418159 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8089298b_3fd9_4bea_a592_525f4a65afdd.slice/crio-5b303835c19251b3a755da79955d558e6835b2e910e9f4f320b612510c1b081a WatchSource:0}: Error finding container 5b303835c19251b3a755da79955d558e6835b2e910e9f4f320b612510c1b081a: Status 404 returned error can't find the container with id 5b303835c19251b3a755da79955d558e6835b2e910e9f4f320b612510c1b081a Nov 03 00:42:38.592144 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:38.592111 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65f5668669-x5rsr" event={"ID":"8089298b-3fd9-4bea-a592-525f4a65afdd","Type":"ContainerStarted","Data":"5b303835c19251b3a755da79955d558e6835b2e910e9f4f320b612510c1b081a"} Nov 03 00:42:39.601064 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:39.600975 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65f5668669-x5rsr" event={"ID":"8089298b-3fd9-4bea-a592-525f4a65afdd","Type":"ContainerStarted","Data":"5259a85bad03260f5be0d3ba70a9f9889c24f74eb9bf17f64c7c55686ea3a404"} Nov 03 00:42:39.601445 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:39.601108 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-65f5668669-x5rsr" Nov 03 00:42:39.711974 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:39.711910 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-65f5668669-x5rsr" podStartSLOduration=1.801529692 podStartE2EDuration="2.711877182s" totalImagesPullingTime="910.34749ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:42:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:42:39.710633613 +0000 UTC m=+6608.679165379" watchObservedRunningTime="2026-11-03 00:42:39.711877182 +0000 UTC m=+6608.680408992" Nov 03 00:42:39.885874 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:42:39.885775 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:42:45.612427 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:45.612349 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-65f5668669-x5rsr" Nov 03 00:42:45.793882 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:45.793848 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-58b7f98cb5-nmrcw"] Nov 03 00:42:45.797851 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:45.797822 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58b7f98cb5-nmrcw" Nov 03 00:42:45.875175 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:45.875093 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-58b7f98cb5-nmrcw"] Nov 03 00:42:45.939484 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:45.939447 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc5bbba-ee70-4f9a-b5bd-04f0c689188e-metrics-certs\") pod \"maas-api-58b7f98cb5-nmrcw\" (UID: \"dfc5bbba-ee70-4f9a-b5bd-04f0c689188e\") " pod="odh-ai-gateway-infra/maas-api-58b7f98cb5-nmrcw" Nov 03 00:42:45.939484 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:45.939489 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfc5bbba-ee70-4f9a-b5bd-04f0c689188e-openshift-service-ca\") pod \"maas-api-58b7f98cb5-nmrcw\" (UID: \"dfc5bbba-ee70-4f9a-b5bd-04f0c689188e\") " pod="odh-ai-gateway-infra/maas-api-58b7f98cb5-nmrcw" Nov 03 00:42:45.939705 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:45.939580 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpcxv\" (UniqueName: \"kubernetes.io/projected/dfc5bbba-ee70-4f9a-b5bd-04f0c689188e-kube-api-access-wpcxv\") pod \"maas-api-58b7f98cb5-nmrcw\" (UID: \"dfc5bbba-ee70-4f9a-b5bd-04f0c689188e\") " pod="odh-ai-gateway-infra/maas-api-58b7f98cb5-nmrcw" Nov 03 00:42:45.939744 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:45.939724 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfc5bbba-ee70-4f9a-b5bd-04f0c689188e-maas-api-tls\") pod \"maas-api-58b7f98cb5-nmrcw\" (UID: \"dfc5bbba-ee70-4f9a-b5bd-04f0c689188e\") " pod="odh-ai-gateway-infra/maas-api-58b7f98cb5-nmrcw" Nov 03 00:42:46.040967 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:46.040925 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc5bbba-ee70-4f9a-b5bd-04f0c689188e-metrics-certs\") pod \"maas-api-58b7f98cb5-nmrcw\" (UID: \"dfc5bbba-ee70-4f9a-b5bd-04f0c689188e\") " pod="odh-ai-gateway-infra/maas-api-58b7f98cb5-nmrcw" Nov 03 00:42:46.041178 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:46.040977 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfc5bbba-ee70-4f9a-b5bd-04f0c689188e-openshift-service-ca\") pod \"maas-api-58b7f98cb5-nmrcw\" (UID: \"dfc5bbba-ee70-4f9a-b5bd-04f0c689188e\") " pod="odh-ai-gateway-infra/maas-api-58b7f98cb5-nmrcw" Nov 03 00:42:46.041178 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:46.041011 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpcxv\" (UniqueName: \"kubernetes.io/projected/dfc5bbba-ee70-4f9a-b5bd-04f0c689188e-kube-api-access-wpcxv\") pod \"maas-api-58b7f98cb5-nmrcw\" (UID: \"dfc5bbba-ee70-4f9a-b5bd-04f0c689188e\") " pod="odh-ai-gateway-infra/maas-api-58b7f98cb5-nmrcw" Nov 03 00:42:46.041178 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:46.041130 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfc5bbba-ee70-4f9a-b5bd-04f0c689188e-maas-api-tls\") pod \"maas-api-58b7f98cb5-nmrcw\" (UID: \"dfc5bbba-ee70-4f9a-b5bd-04f0c689188e\") " pod="odh-ai-gateway-infra/maas-api-58b7f98cb5-nmrcw" Nov 03 00:42:46.041680 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:46.041655 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfc5bbba-ee70-4f9a-b5bd-04f0c689188e-openshift-service-ca\") pod \"maas-api-58b7f98cb5-nmrcw\" (UID: \"dfc5bbba-ee70-4f9a-b5bd-04f0c689188e\") " pod="odh-ai-gateway-infra/maas-api-58b7f98cb5-nmrcw" Nov 03 00:42:46.043515 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:46.043492 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc5bbba-ee70-4f9a-b5bd-04f0c689188e-metrics-certs\") pod \"maas-api-58b7f98cb5-nmrcw\" (UID: \"dfc5bbba-ee70-4f9a-b5bd-04f0c689188e\") " pod="odh-ai-gateway-infra/maas-api-58b7f98cb5-nmrcw" Nov 03 00:42:46.043515 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:46.043509 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfc5bbba-ee70-4f9a-b5bd-04f0c689188e-maas-api-tls\") pod \"maas-api-58b7f98cb5-nmrcw\" (UID: \"dfc5bbba-ee70-4f9a-b5bd-04f0c689188e\") " pod="odh-ai-gateway-infra/maas-api-58b7f98cb5-nmrcw" Nov 03 00:42:46.095803 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:46.095774 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpcxv\" (UniqueName: \"kubernetes.io/projected/dfc5bbba-ee70-4f9a-b5bd-04f0c689188e-kube-api-access-wpcxv\") pod \"maas-api-58b7f98cb5-nmrcw\" (UID: \"dfc5bbba-ee70-4f9a-b5bd-04f0c689188e\") " pod="odh-ai-gateway-infra/maas-api-58b7f98cb5-nmrcw" Nov 03 00:42:46.114751 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:46.114719 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58b7f98cb5-nmrcw" Nov 03 00:42:46.485666 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:46.485636 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-58b7f98cb5-nmrcw"] Nov 03 00:42:46.487314 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:42:46.487286 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc5bbba_ee70_4f9a_b5bd_04f0c689188e.slice/crio-50d923c89169f66c25daab904362703381eed280aa99d7f364ac8739c1861926 WatchSource:0}: Error finding container 50d923c89169f66c25daab904362703381eed280aa99d7f364ac8739c1861926: Status 404 returned error can't find the container with id 50d923c89169f66c25daab904362703381eed280aa99d7f364ac8739c1861926 Nov 03 00:42:46.637828 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:46.637791 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58b7f98cb5-nmrcw" event={"ID":"dfc5bbba-ee70-4f9a-b5bd-04f0c689188e","Type":"ContainerStarted","Data":"50d923c89169f66c25daab904362703381eed280aa99d7f364ac8739c1861926"} Nov 03 00:42:47.644747 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:47.644652 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58b7f98cb5-nmrcw" event={"ID":"dfc5bbba-ee70-4f9a-b5bd-04f0c689188e","Type":"ContainerStarted","Data":"1ea09370a7f77f3b34c1da9586fcf0a052491a6a4a9e26cf5d35d867f9f8fe3b"} Nov 03 00:42:47.645165 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:47.644812 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-58b7f98cb5-nmrcw" Nov 03 00:42:47.796159 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:47.796095 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-58b7f98cb5-nmrcw" podStartSLOduration=1.903961571 podStartE2EDuration="2.796082278s" totalImagesPullingTime="892.120707ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:42:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:42:47.755172992 +0000 UTC m=+6616.723704775" watchObservedRunningTime="2026-11-03 00:42:47.796082278 +0000 UTC m=+6616.764614062" Nov 03 00:42:53.655263 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:53.655229 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-58b7f98cb5-nmrcw" Nov 03 00:42:53.815195 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:53.815161 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-65f5668669-x5rsr"] Nov 03 00:42:53.815412 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:53.815388 5076 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-65f5668669-x5rsr" podUID="8089298b-3fd9-4bea-a592-525f4a65afdd" containerName="maas-api" containerID="cri-o://5259a85bad03260f5be0d3ba70a9f9889c24f74eb9bf17f64c7c55686ea3a404" gracePeriod=30 Nov 03 00:42:54.085075 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:54.085050 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65f5668669-x5rsr" Nov 03 00:42:54.165814 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:54.165781 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8089298b-3fd9-4bea-a592-525f4a65afdd-kube-api-access-cm6bx\" (UniqueName: \"kubernetes.io/projected/8089298b-3fd9-4bea-a592-525f4a65afdd-kube-api-access-cm6bx\") pod \"8089298b-3fd9-4bea-a592-525f4a65afdd\" (UID: \"8089298b-3fd9-4bea-a592-525f4a65afdd\") " Nov 03 00:42:54.166055 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:54.165834 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8089298b-3fd9-4bea-a592-525f4a65afdd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8089298b-3fd9-4bea-a592-525f4a65afdd-openshift-service-ca\") pod \"8089298b-3fd9-4bea-a592-525f4a65afdd\" (UID: \"8089298b-3fd9-4bea-a592-525f4a65afdd\") " Nov 03 00:42:54.166055 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:54.165872 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8089298b-3fd9-4bea-a592-525f4a65afdd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8089298b-3fd9-4bea-a592-525f4a65afdd-maas-api-tls\") pod \"8089298b-3fd9-4bea-a592-525f4a65afdd\" (UID: \"8089298b-3fd9-4bea-a592-525f4a65afdd\") " Nov 03 00:42:54.166055 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:54.165958 5076 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8089298b-3fd9-4bea-a592-525f4a65afdd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8089298b-3fd9-4bea-a592-525f4a65afdd-metrics-certs\") pod \"8089298b-3fd9-4bea-a592-525f4a65afdd\" (UID: \"8089298b-3fd9-4bea-a592-525f4a65afdd\") " Nov 03 00:42:54.166323 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:54.166293 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8089298b-3fd9-4bea-a592-525f4a65afdd-openshift-service-ca" pod "8089298b-3fd9-4bea-a592-525f4a65afdd" (UID: "8089298b-3fd9-4bea-a592-525f4a65afdd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:42:54.168154 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:54.168126 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8089298b-3fd9-4bea-a592-525f4a65afdd-metrics-certs" pod "8089298b-3fd9-4bea-a592-525f4a65afdd" (UID: "8089298b-3fd9-4bea-a592-525f4a65afdd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:42:54.168251 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:54.168164 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8089298b-3fd9-4bea-a592-525f4a65afdd-maas-api-tls" pod "8089298b-3fd9-4bea-a592-525f4a65afdd" (UID: "8089298b-3fd9-4bea-a592-525f4a65afdd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:42:54.168251 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:54.168189 5076 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8089298b-3fd9-4bea-a592-525f4a65afdd-kube-api-access-cm6bx" pod "8089298b-3fd9-4bea-a592-525f4a65afdd" (UID: "8089298b-3fd9-4bea-a592-525f4a65afdd"). InnerVolumeSpecName "kube-api-access-cm6bx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:42:54.267154 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:54.267114 5076 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm6bx\" (UniqueName: \"kubernetes.io/projected/8089298b-3fd9-4bea-a592-525f4a65afdd-kube-api-access-cm6bx\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:42:54.267154 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:54.267146 5076 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8089298b-3fd9-4bea-a592-525f4a65afdd-openshift-service-ca\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:42:54.267154 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:54.267156 5076 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8089298b-3fd9-4bea-a592-525f4a65afdd-maas-api-tls\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:42:54.267154 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:54.267165 5076 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8089298b-3fd9-4bea-a592-525f4a65afdd-metrics-certs\") on node \"ip-10-0-130-211.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:42:54.702270 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:54.702232 5076 generic.go:356] "Generic (PLEG): container finished" podID="8089298b-3fd9-4bea-a592-525f4a65afdd" containerID="5259a85bad03260f5be0d3ba70a9f9889c24f74eb9bf17f64c7c55686ea3a404" exitCode=0 Nov 03 00:42:54.702745 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:54.702282 5076 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65f5668669-x5rsr" Nov 03 00:42:54.702745 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:54.702316 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65f5668669-x5rsr" event={"ID":"8089298b-3fd9-4bea-a592-525f4a65afdd","Type":"ContainerDied","Data":"5259a85bad03260f5be0d3ba70a9f9889c24f74eb9bf17f64c7c55686ea3a404"} Nov 03 00:42:54.702745 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:54.702375 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65f5668669-x5rsr" event={"ID":"8089298b-3fd9-4bea-a592-525f4a65afdd","Type":"ContainerDied","Data":"5b303835c19251b3a755da79955d558e6835b2e910e9f4f320b612510c1b081a"} Nov 03 00:42:54.702745 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:54.702404 5076 scope.go:122] "RemoveContainer" containerID="5259a85bad03260f5be0d3ba70a9f9889c24f74eb9bf17f64c7c55686ea3a404" Nov 03 00:42:54.712146 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:54.712130 5076 scope.go:122] "RemoveContainer" containerID="5259a85bad03260f5be0d3ba70a9f9889c24f74eb9bf17f64c7c55686ea3a404" Nov 03 00:42:54.712418 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:42:54.712394 5076 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5259a85bad03260f5be0d3ba70a9f9889c24f74eb9bf17f64c7c55686ea3a404\": container with ID starting with 5259a85bad03260f5be0d3ba70a9f9889c24f74eb9bf17f64c7c55686ea3a404 not found: ID does not exist" containerID="5259a85bad03260f5be0d3ba70a9f9889c24f74eb9bf17f64c7c55686ea3a404" Nov 03 00:42:54.712485 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:54.712431 5076 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5259a85bad03260f5be0d3ba70a9f9889c24f74eb9bf17f64c7c55686ea3a404"} err="failed to get container status \"5259a85bad03260f5be0d3ba70a9f9889c24f74eb9bf17f64c7c55686ea3a404\": rpc error: code = NotFound desc = could not find container \"5259a85bad03260f5be0d3ba70a9f9889c24f74eb9bf17f64c7c55686ea3a404\": container with ID starting with 5259a85bad03260f5be0d3ba70a9f9889c24f74eb9bf17f64c7c55686ea3a404 not found: ID does not exist" Nov 03 00:42:54.859154 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:54.859119 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-65f5668669-x5rsr"] Nov 03 00:42:54.906490 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:54.906443 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-65f5668669-x5rsr"] Nov 03 00:42:55.150749 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:42:55.150656 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8089298b-3fd9-4bea-a592-525f4a65afdd" path="/var/lib/kubelet/pods/8089298b-3fd9-4bea-a592-525f4a65afdd/volumes" Nov 03 00:43:09.884289 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:43:09.884245 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:43:33.620177 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:43:33.620143 5076 scope.go:122] "RemoveContainer" containerID="fe2327dce914d6b1a9a676ed95df3b7da5e5e5d56ec22be3766d2c52235b7b3c" Nov 03 00:43:33.656947 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:43:33.648639 5076 scope.go:122] "RemoveContainer" containerID="cbb3a609868c7d0831abcf3506f7fcc4eff82f2c0074287de812cb3a8e957795" Nov 03 00:43:33.704127 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:43:33.704097 5076 scope.go:122] "RemoveContainer" containerID="dddb10365ae1884a87611bd20df6c8cb1b06bd8b47d9aec28aa29b6ce4fb7eea" Nov 03 00:43:39.884186 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:43:39.884146 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:44:09.882502 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:44:09.882460 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:44:33.868172 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:44:33.868083 5076 scope.go:122] "RemoveContainer" containerID="bf576ee9546bb98e2626a9d5ce7a69f9a86f14ac461d21b4f88d87449ecb6b4e" Nov 03 00:44:33.877556 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:44:33.877533 5076 scope.go:122] "RemoveContainer" containerID="906b7c4409ccf54850d6cf474171b7f747c4a3669732da9ba000d5ae33d52bee" Nov 03 00:44:39.883753 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:44:39.883715 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:45:01.458914 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:45:01.458049 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gctbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:45:01.458914 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:45:01.458136 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" podUID="d6db6fe8-b76c-4878-b0f3-bd347b2d0cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:45:09.883320 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:45:09.883271 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:45:18.057919 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:45:18.050160 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjtqd"] Nov 03 00:45:18.104633 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:45:18.104563 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjtqd"] Nov 03 00:45:19.182919 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:45:19.177570 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd30c2f2-b0ab-42e1-affa-c5ec030f8121" path="/var/lib/kubelet/pods/fd30c2f2-b0ab-42e1-affa-c5ec030f8121/volumes" Nov 03 00:45:39.883443 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:45:39.883395 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:46:09.884544 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:46:09.884494 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:46:33.908730 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:46:33.908695 5076 scope.go:122] "RemoveContainer" containerID="d85eba31a9eda17d2a54ab259d2446f66e7792593f048778831c2e212a844a60" Nov 03 00:46:39.882193 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:46:39.882156 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:46:46.463155 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:46:46.463113 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gctbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:46:46.464018 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:46:46.463982 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" podUID="d6db6fe8-b76c-4878-b0f3-bd347b2d0cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:46:46.817971 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:46:46.814501 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-784vq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:46:46.817971 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:46:46.814569 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" podUID="506655af-c50a-4207-906e-9f705fc8f9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:47:07.055914 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:47:07.055818 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-m9q7q"] Nov 03 00:47:07.101912 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:47:07.101851 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-m9q7q"] Nov 03 00:47:07.166535 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:47:07.166498 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd7427ec-e02e-4e23-ae32-27ae6ddfbad4" path="/var/lib/kubelet/pods/fd7427ec-e02e-4e23-ae32-27ae6ddfbad4/volumes" Nov 03 00:47:09.883841 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:47:09.883802 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:47:28.051028 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:47:28.046730 5076 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4vvkh"] Nov 03 00:47:28.094151 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:47:28.094108 5076 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4vvkh"] Nov 03 00:47:29.185916 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:47:29.180167 5076 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="578177a7-bb81-45fc-a26c-d99a99def956" path="/var/lib/kubelet/pods/578177a7-bb81-45fc-a26c-d99a99def956/volumes" Nov 03 00:47:32.395483 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:47:32.395453 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 03 00:47:32.492271 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:47:32.492244 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c2gxc_9c4b6fbd-74cf-41c6-bbae-92748ac25e1e/console-operator/2.log" Nov 03 00:47:39.883876 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:47:39.883834 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:48:01.467914 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:48:01.467009 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gctbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:48:01.467914 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:48:01.467123 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gctbb" podUID="d6db6fe8-b76c-4878-b0f3-bd347b2d0cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:48:01.822973 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:48:01.822002 5076 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-784vq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:48:01.822973 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:48:01.822090 5076 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-784vq" podUID="506655af-c50a-4207-906e-9f705fc8f9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:48:09.900787 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:48:09.900739 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:48:33.941213 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:48:33.941180 5076 scope.go:122] "RemoveContainer" containerID="bb00b7ca054f78639cd56bf5174bbcf92d6bae6a4a811c5b41da111afa13173f" Nov 03 00:48:33.950376 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:48:33.950355 5076 scope.go:122] "RemoveContainer" containerID="041980d4ec32200b8c503b1b34f1cae8aecc3199d25a6d5b8f7dc92b1e861662" Nov 03 00:48:39.883177 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:48:39.883117 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:49:09.884266 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:49:09.884228 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:49:13.832855 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:49:13.832820 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-hc5jd_5c78102f-a925-4cb0-bc69-971c0d248e4e/manager/0.log" Nov 03 00:49:14.702548 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:49:14.702513 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-b67978596-lth4c_b54d7b4b-cee5-47fc-a662-1b24a9d96356/manager/0.log" Nov 03 00:49:16.635726 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:49:16.635669 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-58b7f98cb5-nmrcw_dfc5bbba-ee70-4f9a-b5bd-04f0c689188e/maas-api/0.log" Nov 03 00:49:20.771801 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:49:20.771764 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-4tbpp_818b5c39-d141-455e-9ff2-ff03b4ecc69d/postgres/0.log" Nov 03 00:49:23.135361 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:49:23.135331 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-5rw74_a90148ca-bc21-4a39-8be9-4cff6cc9b34c/manager/0.log" Nov 03 00:49:23.464329 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:49:23.464302 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-6ff5j_0d983ba6-e119-4cbc-98d2-3dc824673324/manager/0.log" Nov 03 00:49:24.683682 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:49:24.683652 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-4fvc5_c7e772da-ebf0-4dca-ac74-0c8e4cea112e/manager/0.log" Nov 03 00:49:25.842162 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:49:25.842131 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68dd7b898ft527v_65a56e85-82db-48c6-8060-3279887a6e89/istio-proxy/0.log" Nov 03 00:49:26.120484 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:49:26.120400 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-trmks_6f68c4de-4657-4917-a910-72f4b56b39e3/discovery/0.log" Nov 03 00:49:27.428418 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:49:27.428382 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-5xx2r_9cd92df4-350d-4fe2-bd41-48b57198a268/istio-proxy/0.log" Nov 03 00:49:27.744561 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:49:27.744529 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-784vq_506655af-c50a-4207-906e-9f705fc8f9f6/istio-proxy/0.log" Nov 03 00:49:28.306921 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:49:28.306879 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-gctbb_d6db6fe8-b76c-4878-b0f3-bd347b2d0cde/istio-proxy/0.log" Nov 03 00:49:29.310315 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:49:29.310281 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-slv2t_4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4/istio-proxy/0.log" Nov 03 00:49:29.582355 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:49:29.582286 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg_c76adda1-d0ff-456d-a420-24babb08880b/istio-proxy/0.log" Nov 03 00:49:30.235413 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:49:30.235377 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-896ccfb6c-q4rg5_75f7d1d7-88e4-41e0-bdd3-a43df363b408/router/0.log" Nov 03 00:49:32.280394 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:49:32.280361 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wrqhh_00417fb1-0bb2-48cf-85a4-deca8e7f01a5/main/0.log" Nov 03 00:49:32.724593 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:49:32.724560 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-khjtw_5a0c3861-1c5a-4155-ae4b-785129cd0f1e/main/0.log" Nov 03 00:49:39.883327 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:49:39.883285 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:49:54.510155 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:49:54.510108 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-k4gfs_fd96552a-1a78-4fac-8faf-71d36f010905/global-pull-secret-syncer/0.log" Nov 03 00:49:55.137023 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:49:55.136993 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-fdj8t_42d598be-a6f1-486e-9ac3-2f5ece575f89/connection-checker/0.log" Nov 03 00:49:55.558134 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:49:55.558105 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-mtrcx_4b9cd8c7-228b-451a-afe4-3af7a3bf0a42/connection-checker/0.log" Nov 03 00:49:55.991674 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:49:55.991643 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-pxpxp_fbcab4d0-9dd6-4682-be01-c4f589697e95/connection-checker/0.log" Nov 03 00:49:57.123103 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:49:57.123065 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-hc5jd_5c78102f-a925-4cb0-bc69-971c0d248e4e/manager/0.log" Nov 03 00:49:57.376426 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:49:57.376344 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zqssw_8fbcc658-b377-4da4-bf1d-e6274dfd805b/konnectivity-agent/0.log" Nov 03 00:49:57.762617 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:49:57.762583 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-211.us-west-2.compute.internal_7ba3fed7e8adcc2e1f404604f948f337/haproxy/0.log" Nov 03 00:49:58.261580 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:49:58.261545 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-b67978596-lth4c_b54d7b4b-cee5-47fc-a662-1b24a9d96356/manager/0.log" Nov 03 00:50:04.126382 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:04.126350 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-hc5jd_5c78102f-a925-4cb0-bc69-971c0d248e4e/manager/0.log" Nov 03 00:50:05.010405 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:05.010361 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-5rw74_a90148ca-bc21-4a39-8be9-4cff6cc9b34c/manager/0.log" Nov 03 00:50:05.280102 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:05.280020 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-b67978596-lth4c_b54d7b4b-cee5-47fc-a662-1b24a9d96356/manager/0.log" Nov 03 00:50:05.424116 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:05.424087 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-6ff5j_0d983ba6-e119-4cbc-98d2-3dc824673324/manager/0.log" Nov 03 00:50:05.665658 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:05.665581 5076 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cgjxh/perf-node-gather-daemonset-ld67k"] Nov 03 00:50:05.666062 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:05.666047 5076 cpu_manager.go:414] "containerMap: removing container" podUID="8089298b-3fd9-4bea-a592-525f4a65afdd" containerName="maas-api" Nov 03 00:50:05.666117 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:05.666063 5076 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8089298b-3fd9-4bea-a592-525f4a65afdd" containerName="maas-api" Nov 03 00:50:05.666168 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:05.666159 5076 memory_manager.go:365] "RemoveStaleState removing state" podUID="8089298b-3fd9-4bea-a592-525f4a65afdd" containerName="maas-api" Nov 03 00:50:05.669236 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:05.669219 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-ld67k" Nov 03 00:50:05.719588 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:05.719545 5076 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cgjxh\"/\"default-dockercfg-hnhxn\"" Nov 03 00:50:05.759628 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:05.759590 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/69e6657a-5d22-4e55-b74b-3dfd1185a504-lib-modules\") pod \"perf-node-gather-daemonset-ld67k\" (UID: \"69e6657a-5d22-4e55-b74b-3dfd1185a504\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-ld67k" Nov 03 00:50:05.759935 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:05.759651 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb89k\" (UniqueName: \"kubernetes.io/projected/69e6657a-5d22-4e55-b74b-3dfd1185a504-kube-api-access-tb89k\") pod \"perf-node-gather-daemonset-ld67k\" (UID: \"69e6657a-5d22-4e55-b74b-3dfd1185a504\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-ld67k" Nov 03 00:50:05.759935 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:05.759788 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/69e6657a-5d22-4e55-b74b-3dfd1185a504-proc\") pod \"perf-node-gather-daemonset-ld67k\" (UID: \"69e6657a-5d22-4e55-b74b-3dfd1185a504\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-ld67k" Nov 03 00:50:05.760112 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:05.759945 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/69e6657a-5d22-4e55-b74b-3dfd1185a504-podres\") pod \"perf-node-gather-daemonset-ld67k\" (UID: \"69e6657a-5d22-4e55-b74b-3dfd1185a504\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-ld67k" Nov 03 00:50:05.760112 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:05.760064 5076 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/69e6657a-5d22-4e55-b74b-3dfd1185a504-sys\") pod \"perf-node-gather-daemonset-ld67k\" (UID: \"69e6657a-5d22-4e55-b74b-3dfd1185a504\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-ld67k" Nov 03 00:50:05.766920 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:05.766879 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cgjxh\"/\"openshift-service-ca.crt\"" Nov 03 00:50:05.767079 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:05.767061 5076 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cgjxh\"/\"kube-root-ca.crt\"" Nov 03 00:50:05.769350 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:05.769317 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cgjxh/perf-node-gather-daemonset-ld67k"] Nov 03 00:50:05.861207 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:05.861175 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/69e6657a-5d22-4e55-b74b-3dfd1185a504-lib-modules\") pod \"perf-node-gather-daemonset-ld67k\" (UID: \"69e6657a-5d22-4e55-b74b-3dfd1185a504\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-ld67k" Nov 03 00:50:05.861430 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:05.861217 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb89k\" (UniqueName: \"kubernetes.io/projected/69e6657a-5d22-4e55-b74b-3dfd1185a504-kube-api-access-tb89k\") pod \"perf-node-gather-daemonset-ld67k\" (UID: \"69e6657a-5d22-4e55-b74b-3dfd1185a504\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-ld67k" Nov 03 00:50:05.861430 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:05.861286 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/69e6657a-5d22-4e55-b74b-3dfd1185a504-proc\") pod \"perf-node-gather-daemonset-ld67k\" (UID: \"69e6657a-5d22-4e55-b74b-3dfd1185a504\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-ld67k" Nov 03 00:50:05.861430 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:05.861320 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/69e6657a-5d22-4e55-b74b-3dfd1185a504-lib-modules\") pod \"perf-node-gather-daemonset-ld67k\" (UID: \"69e6657a-5d22-4e55-b74b-3dfd1185a504\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-ld67k" Nov 03 00:50:05.861430 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:05.861333 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/69e6657a-5d22-4e55-b74b-3dfd1185a504-podres\") pod \"perf-node-gather-daemonset-ld67k\" (UID: \"69e6657a-5d22-4e55-b74b-3dfd1185a504\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-ld67k" Nov 03 00:50:05.861430 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:05.861387 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/69e6657a-5d22-4e55-b74b-3dfd1185a504-proc\") pod \"perf-node-gather-daemonset-ld67k\" (UID: \"69e6657a-5d22-4e55-b74b-3dfd1185a504\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-ld67k" Nov 03 00:50:05.861430 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:05.861394 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/69e6657a-5d22-4e55-b74b-3dfd1185a504-podres\") pod \"perf-node-gather-daemonset-ld67k\" (UID: \"69e6657a-5d22-4e55-b74b-3dfd1185a504\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-ld67k" Nov 03 00:50:05.861430 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:05.861405 5076 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/69e6657a-5d22-4e55-b74b-3dfd1185a504-sys\") pod \"perf-node-gather-daemonset-ld67k\" (UID: \"69e6657a-5d22-4e55-b74b-3dfd1185a504\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-ld67k" Nov 03 00:50:05.861762 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:05.861445 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/69e6657a-5d22-4e55-b74b-3dfd1185a504-sys\") pod \"perf-node-gather-daemonset-ld67k\" (UID: \"69e6657a-5d22-4e55-b74b-3dfd1185a504\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-ld67k" Nov 03 00:50:05.916456 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:05.916383 5076 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb89k\" (UniqueName: \"kubernetes.io/projected/69e6657a-5d22-4e55-b74b-3dfd1185a504-kube-api-access-tb89k\") pod \"perf-node-gather-daemonset-ld67k\" (UID: \"69e6657a-5d22-4e55-b74b-3dfd1185a504\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-ld67k" Nov 03 00:50:05.981950 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:05.981919 5076 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-ld67k" Nov 03 00:50:06.347663 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:06.347627 5076 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cgjxh/perf-node-gather-daemonset-ld67k"] Nov 03 00:50:06.349054 ip-10-0-130-211 kubenswrapper[5076]: W1103 00:50:06.349025 5076 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod69e6657a_5d22_4e55_b74b_3dfd1185a504.slice/crio-1900cd2bd3e1b96bf70ddeb02a7df5df0d228814de8b681214bf5fbd7594f4dc WatchSource:0}: Error finding container 1900cd2bd3e1b96bf70ddeb02a7df5df0d228814de8b681214bf5fbd7594f4dc: Status 404 returned error can't find the container with id 1900cd2bd3e1b96bf70ddeb02a7df5df0d228814de8b681214bf5fbd7594f4dc Nov 03 00:50:06.834316 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:06.834284 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-ld67k" event={"ID":"69e6657a-5d22-4e55-b74b-3dfd1185a504","Type":"ContainerStarted","Data":"c94575678ca65274ff391a17bdd3deb681a3c2f0dccf547484b0077363d37f93"} Nov 03 00:50:06.834510 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:06.834333 5076 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-ld67k" event={"ID":"69e6657a-5d22-4e55-b74b-3dfd1185a504","Type":"ContainerStarted","Data":"1900cd2bd3e1b96bf70ddeb02a7df5df0d228814de8b681214bf5fbd7594f4dc"} Nov 03 00:50:06.834510 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:06.834389 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-ld67k" Nov 03 00:50:06.944286 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:06.944233 5076 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-ld67k" podStartSLOduration=1.9442184710000001 podStartE2EDuration="1.944218471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:06.942245591 +0000 UTC m=+7055.910777375" watchObservedRunningTime="2026-11-03 00:50:06.944218471 +0000 UTC m=+7055.912750255" Nov 03 00:50:07.194903 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:07.194870 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-4fvc5_c7e772da-ebf0-4dca-ac74-0c8e4cea112e/manager/0.log" Nov 03 00:50:09.883775 ip-10-0-130-211 kubenswrapper[5076]: E1103 00:50:09.883738 5076 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sv7kh" containerName="extractor" Nov 03 00:50:10.643545 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:10.643510 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-hc5jd_5c78102f-a925-4cb0-bc69-971c0d248e4e/manager/0.log" Nov 03 00:50:11.903540 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:11.903442 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-b67978596-lth4c_b54d7b4b-cee5-47fc-a662-1b24a9d96356/manager/0.log" Nov 03 00:50:12.562000 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:12.561959 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-mbmf9_15789a0d-e030-45e0-a129-41de738ef767/cluster-monitoring-operator/0.log" Nov 03 00:50:12.850268 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:12.850200 5076 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-ld67k" Nov 03 00:50:17.249956 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:17.249918 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68dd7b898ft527v_65a56e85-82db-48c6-8060-3279887a6e89/istio-proxy/0.log" Nov 03 00:50:17.497953 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:17.497913 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kf24r_14242834-56aa-458b-bacc-bb5a4aa97937/node-exporter/0.log" Nov 03 00:50:18.140276 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:18.140242 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kf24r_14242834-56aa-458b-bacc-bb5a4aa97937/kube-rbac-proxy/0.log" Nov 03 00:50:18.555864 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:18.555838 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kf24r_14242834-56aa-458b-bacc-bb5a4aa97937/init-textfile/0.log" Nov 03 00:50:18.756403 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:18.756369 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-trmks_6f68c4de-4657-4917-a910-72f4b56b39e3/discovery/0.log" Nov 03 00:50:19.713035 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:19.713005 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-5xx2r_9cd92df4-350d-4fe2-bd41-48b57198a268/istio-proxy/0.log" Nov 03 00:50:19.929724 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:19.929674 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-784vq_506655af-c50a-4207-906e-9f705fc8f9f6/istio-proxy/0.log" Nov 03 00:50:20.351767 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:20.351738 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-gctbb_d6db6fe8-b76c-4878-b0f3-bd347b2d0cde/istio-proxy/0.log" Nov 03 00:50:21.197731 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:21.197703 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-slv2t_4b14bc8b-9e90-4e86-b679-a16fbf1f9ba4/istio-proxy/0.log" Nov 03 00:50:21.408186 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:21.408157 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6bd8768855-s8qgg_c76adda1-d0ff-456d-a420-24babb08880b/istio-proxy/0.log" Nov 03 00:50:22.045922 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:22.045877 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-896ccfb6c-q4rg5_75f7d1d7-88e4-41e0-bdd3-a43df363b408/router/0.log" Nov 03 00:50:23.881213 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:23.881181 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-wtk4j_be9397ed-f3c5-4b5d-a300-04f04f8fd355/prometheus-operator-admission-webhook/0.log" Nov 03 00:50:27.688166 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:27.688135 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-5rw74_a90148ca-bc21-4a39-8be9-4cff6cc9b34c/manager/0.log" Nov 03 00:50:27.900104 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:27.900074 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-6ff5j_0d983ba6-e119-4cbc-98d2-3dc824673324/manager/0.log" Nov 03 00:50:28.772948 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:28.772915 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68dd7b898ft527v_65a56e85-82db-48c6-8060-3279887a6e89/istio-proxy/0.log" Nov 03 00:50:28.801152 ip-10-0-130-211 kubenswrapper[5076]: I1103 00:50:28.801111 5076 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-4fvc5_c7e772da-ebf0-4dca-ac74-0c8e4cea112e/manager/0.log"